Video encoding method and apparatus, device, and computer storage medium
By estimating affine motion and iteratively optimizing the motion vector of control points, the problem of high computational cost in existing video coding technologies is solved, improving the efficiency and transmission speed of video coding and reducing video playback stuttering.
Patent Information
- Application Number
- PCT/CN2026/095088
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-16
- Filing Date
- 2026-05-07
- Publication Date
- 2026-11-19
AI Technical Summary
Existing video coding technologies have high computational costs in determining motion vectors, resulting in low video coding efficiency, especially in live streaming scenarios where video transmission is prone to stuttering.
An affine motion estimation method is adopted. By acquiring the motion vectors of control points in video frames and performing iterative optimization, the motion vectors are adjusted using gradient information to reduce prediction residuals and improve the accuracy and efficiency of motion compensation.
By iteratively optimizing motion vectors through multiple rounds, the computational resource consumption of video encoding is reduced, the compression efficiency and transmission speed of video encoding are improved, and the stuttering phenomenon in video playback is reduced.
Smart Images

Figure CN2026095088_19112026_PF_FP_ABST
Abstract
Description
Video encoding methods, apparatus and equipment and computer storage media
[0001] This application claims priority to Chinese Patent Application No. 2025106439449, filed on May 16, 2025, entitled “A video coding method, apparatus and device and computer storage medium”, the entire contents of which are incorporated herein by reference. Technical Field
[0002] This application relates to the field of computer technology, and in particular to a video encoding method, apparatus and device, and computer storage medium. Background Technology
[0003] Video coding technology refers to the process of converting raw video signals into compressed digital formats using coding algorithms, thereby significantly reducing the amount of video data while ensuring visual quality.
[0004] In practical applications, the motion changes of an object between multiple consecutive video frames in a video recording can be represented by motion vectors. Referring to Figure 1A, when encoding a video recording, the recording can be compressed into a reference frame and at least one prediction frame associated with the reference frame. All pixel information in the reference frame is recorded, while pixel information in other prediction frames can be described by the motion vectors of the object between the reference and prediction frames and the corresponding residual information (i.e., prediction residuals).
[0005] Correspondingly, in the video decoding process, for a prediction frame, motion compensation processing is performed using the pixel information of the reference frame and the motion vector corresponding to the prediction frame to obtain the prediction result for the prediction frame. Then, the residual information is superimposed on the prediction result to decode the pixel information in the prediction frame.
[0006] In the above process, only the pixel information of the reference frame needs to be recorded, while the pixel information of other predicted frames can be encoded into corresponding motion vectors and residual information. Therefore, the amount of data in the encoded video data can be greatly reduced.
[0007] It is evident that the selection of motion vectors is a crucial step in video coding technology. The motion of an object can take many complex forms, such as flipping and rotating, requiring affine motion estimation to determine the corresponding motion vectors.
[0008] There is an urgent need for a new video coding method to select motion vectors, in order to reduce the computational cost of video coding and thus improve the video coding rate. Summary of the Invention
[0009] This application provides a video encoding method, apparatus, device, and computer storage medium to reduce the computational cost of video encoding, thereby improving the efficiency of video encoding.
[0010] Firstly, a video encoding method is provided, including:
[0011] Obtain the control point motion vectors of at least two control points in the current block of a video frame; each control point motion vector represents the motion vector of the corresponding control point between the video frame and the reference frame;
[0012] The motion vector of the control point is optimized through at least one round of iterations; wherein each round of iterations includes:
[0013] Based on the reference frame and the motion vector of the current control point, determine the prediction residual of the current block;
[0014] Determine the gradient of each component of the predicted residual relative to the motion vector of the current control point; update the motion vector of the current control point based on the gradient and the predicted residual.
[0015] Secondly, a video encoding apparatus is provided, comprising:
[0016] The acquisition module is used to acquire the control point motion vectors of at least two control points of the current block in a video frame; each control point motion vector represents the motion vector of the corresponding control point between the video frame and the reference frame;
[0017] The processing module is used to perform at least one round of iterative optimization on the motion vector of the control point; wherein each round of iterative optimization includes:
[0018] Based on the reference frame and the motion vector of the current control point, determine the prediction residual of the current block;
[0019] Determine the gradient of the predicted residual with respect to each component of the motion vector of the current control point;
[0020] The motion vector of the current control point is updated based on the gradient and the prediction residual.
[0021] Thirdly, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described in the first aspect.
[0022] Fourthly, a computer device is provided, comprising:
[0023] Memory, used to store program instructions;
[0024] A processor is configured to invoke program instructions stored in the memory and execute the method described in the first aspect according to the obtained program instructions.
[0025] Fifthly, a computer-readable storage medium is provided, the computer-readable storage medium storing computer-executable instructions for causing a computer to perform the method as described in the first aspect. Attached Figure Description
[0026] Figure 1A is a schematic diagram of a scenario for video coding methods in related technologies;
[0027] Figure 1B is a schematic diagram of a scenario for video coding methods in related technologies.
[0028] Figure 1C illustrates an application scenario of the video encoding method provided in this embodiment.
[0029] Figure 2 is a flowchart illustrating a video encoding method provided in an embodiment of this application.
[0030] Figure 3A is a schematic diagram of a video encoding method provided in an embodiment of this application.
[0031] Figure 3B is a schematic diagram of a video encoding method provided in an embodiment of this application.
[0032] Figure 3C is a schematic diagram of a video encoding method provided in an embodiment of this application.
[0033] Figure 4A is a schematic diagram of a video encoding method provided in an embodiment of this application.
[0034] Figure 4B is a schematic diagram of a video encoding method provided in an embodiment of this application.
[0035] Figure 4C is a schematic diagram of a video encoding method provided in an embodiment of this application.
[0036] Figure 5A is a schematic diagram of a video encoding method provided in an embodiment of this application.
[0037] Figure 5B is a schematic diagram of a video encoding method provided in an embodiment of this application.
[0038] Figure 5C is a schematic diagram of a video encoding method provided in an embodiment of this application.
[0039] Figure 5D is a schematic diagram of a video encoding method provided in an embodiment of this application.
[0040] Figure 5E is a schematic diagram of a video encoding method provided in an embodiment of this application.
[0041] Figure 5F is a schematic diagram of a video encoding method provided in an embodiment of this application.
[0042] Figure 6A is a schematic diagram of a video encoding method provided in an embodiment of this application.
[0043] Figure 6B is a schematic diagram of a video encoding method provided in an embodiment of this application.
[0044] Figure 6C is a schematic diagram of a video encoding method provided in an embodiment of this application.
[0045] Figure 7 is a schematic diagram of a video encoding method provided in an embodiment of this application.
[0046] Figure 8 is a schematic diagram of a video encoding device provided in an embodiment of this application;
[0047] Figure 9 is a schematic diagram of a video encoding device provided in an embodiment of this application. Detailed Implementation
[0048] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.
[0049] The following explanations of some terms used in the embodiments of this application are provided to facilitate understanding by those skilled in the art.
[0050] (1) Video encoding:
[0051] Video coding is the process of reducing video data size through compression techniques, with its core being the elimination of temporal, spatial, and statistical redundancy. Key technologies in video coding include intra-frame prediction (utilizing the spatial correlation between adjacent pixels) and inter-frame prediction (eliminating repetitive information between consecutive frames through motion estimation and compensation). For video coding technology, established coding standards define compression algorithms and bitstream formats to balance compression efficiency and video quality, and are widely used in streaming media, video conferencing, digital television, and other fields.
[0052] (2) Affine motion estimation:
[0053] Affine motion estimation is a predictive technique used in video coding to describe complex motion models. Compared to traditional translational motion estimation (which only considers translation vectors), affine motion estimation more accurately depicts the deformation or perspective changes of objects in consecutive frames through affine transformations (such as rotation, scaling, and shearing).
[0054] Specifically, affine motion estimation constructs a motion vector field using a 6-parameter affine model (two translations + four deformation parameters) or a 4-parameter affine model (translation, rotation, scaling). It then uses the motion vectors of adjacent blocks in the prediction frame to predict the motion of the current block, reducing the amount of residual data. This technique is particularly suitable for handling irregular motions (such as cloud diffusion, water ripples, and object rotation), and can significantly improve compression efficiency.
[0055] (3) Affine model:
[0056] Affine models are mathematical models based on affine transformations used to describe linear geometric transformations in two-dimensional or three-dimensional space, including translation, rotation, scaling, shearing, and combinations thereof. In video coding, affine models more accurately fit complex motions (such as deformation, perspective changes, or non-rigid motion) of objects in consecutive video frames by parameterizing motion vector fields (e.g., a 6-parameter model: translations in the x and y directions, rotation angles, horizontal and vertical scaling factors, and shearing factors) or a simplified 4-parameter model (translation, rotation, and uniform scaling). Compared to traditional translational motion models (which only describe the overall displacement of the object), affine models can capture local deformations or global perspective changes, reducing prediction residuals and improving compression efficiency through more accurate motion compensation.
[0057] (4) Exercise compensation:
[0058] Motion compensation is a key technique for compressing data by leveraging the temporal correlation between adjacent frames. Its core idea is to find the best-matching position of an image patch (or region) in the current frame within a reference frame (such as the previous or next frame) through motion estimation, and to represent this displacement relationship using motion vectors (parameters describing the positional offset). Thus, the encoded video data only needs to record the motion vectors and the difference between the current block and the reference block (i.e., the prediction residual), rather than the complete pixel data, thereby significantly reducing the data volume.
[0059] It should be noted that the embodiments of this application involve operations such as acquiring data video and determining prediction frames and reference frames. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, when relevant data is needed, relevant volunteers can be recruited and relevant agreements authorizing the volunteers to authorize data can be signed, and then the data of these volunteers can be used for implementation; or, implementation can be carried out within the authorized scope of an organization, and the following implementation methods can be used to identify internal members by using the data of internal members; or, the relevant data used in the specific implementation are all simulated data, such as simulated data generated in a virtual scene.
[0060] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.
[0061] The application areas of the video encoding method provided in the embodiments of this application will be briefly introduced below.
[0062] With the continuous development and popularization of technology, more and more people are accustomed to recording their daily lives through video recordings, or sharing their knowledge and experiences. At the same time, video recording technology is becoming increasingly integrated into various aspects of people's work and life. Therefore, after recording specific scenes using hardware devices, video encoding technology has emerged to improve the storage efficiency of video recordings.
[0063] Video coding technology mainly uses coding algorithms to convert the original video signal into a compressed digital format, thereby significantly reducing the size of the encoded video data while ensuring visual quality.
[0064] For example, as shown in Figure 1A, when encoding video recordings, one video frame in the video recording can be selected as a reference frame, and other adjacent video frames can be used as multiple prediction frames that are key to the reference frame.
[0065] In this way, by using the pixel information recorded in the reference frame, as well as the motion vectors and residual information between the reference frame and the prediction frame, the pixel information in other prediction frames can be described. Specifically, during video decoding, for the pixel information of a prediction frame, motion compensation can be performed using the pixel information of the reference frame and the corresponding motion vector of the prediction frame to obtain the prediction result for that prediction frame. Then, the residual information is superimposed on the prediction result to decode the pixel information in the prediction frame.
[0066] In the above process, only the pixel information of the reference frame needs to be recorded, while the pixel information of other predicted frames can be encoded into corresponding motion vectors and residual information. Therefore, the amount of data in the encoded video data can be greatly reduced.
[0067] It is evident that the selection of motion vectors is a crucial step in video coding technology. The motion of an object can take many complex forms, such as flipping and rotating, requiring affine motion estimation to determine the corresponding motion vectors.
[0068] In affine motion estimation, the determination of motion vectors is divided into: the determination of the motion vector of the target point (i.e., the control point motion vector) and the determination of the motion vectors of other points. Referring to Figure 1B, within the target region divided in the prediction frame to be encoded, the target point (i.e., the control point) can be the upper left and upper right corner points. The motion vector of the target point represents the displacement of that point from the reference frame to the current prediction frame. The corresponding motion vectors are mv0 and mv1 marked in Figure 1B. Using the motion vectors corresponding to these two points, as well as the width and height of the region, the motion vectors of other points within the region can be calculated. Thus, based on the motion vectors of each point, the corresponding prediction results can be generated through motion compensation.
[0069] In the above process, determining the motion vector of the target point is the foundation of video coding. Under related technologies, the motion vector is usually determined in the following way: after selecting an initial motion vector for each target point, based on the residual information obtained from each initial motion vector, each initial motion vector is optimized and iterated to reduce the gap between the prediction result and the corresponding prediction frame. In this way, after multiple rounds of iteration, the motion vectors of each target point that meet the conditions can be obtained.
[0070] However, in each round of optimization iteration, Gaussian elimination is typically used to calculate equations for the corresponding residual and gradient information, thereby obtaining the adjustment amount for the motion vector to reduce the residual information between the prediction result and the prediction frame. This calculation method involves a large number of division operations when performing Gaussian elimination on the gradient information, which consumes significant computing resources in hardware implementation. Furthermore, the large amount of data involved in the division operations and the long time required for each operation significantly impact the timing constraints of the hardware, making it difficult for the hardware to quickly and accurately complete the optimization iteration of the motion vector for the target point. Consequently, when video recording is completed and needs to be transmitted to other devices, the inability to quickly encode the video recording results in a prolonged transmission process. For example, in a live streaming scenario, because the captured video recording cannot be quickly encoded and transmitted, noticeable stuttering occurs during playback.
[0071] In view of this, embodiments of this application provide a video encoding method in which corresponding encoding processing is performed on each predicted frame during the video encoding process to improve video encoding efficiency.
[0072] Specifically, when encoding any one of the prediction frames (i.e., the currently predicted video frame) from the key prediction frames of the reference frame, it is first necessary to obtain the initial motion vectors (i.e., initial control point motion vectors) of each target point (i.e., each control point) in a video frame between a prediction frame and the reference frame. For example, obtain the control point motion vectors of at least two control points in the current block of a video frame; each control point motion vector represents the motion vector of the corresponding control point between the video frame and the reference frame. Thus, after obtaining the initial motion vectors corresponding to each target point, iterative optimization can be performed on these initial motion vectors until the resulting output motion vectors meet the preset optimization conditions.
[0073] Specifically, the following operations are performed during each round of optimization:
[0074] Based on the reference frame and the motion vectors output from each previous round, prediction is performed on each pixel in the current block to obtain the prediction result (i.e., the reconstructed pixels of the reference block for the current block), and the difference information (e.g., prediction residual) between the original pixel value of the current block and the prediction result of the current block is determined. For example, the prediction residual of the current block is determined based on the reference frame and the current control point motion vector. The motion vectors output from each previous round can also be called the current control point motion vector. During the first round of optimization, the motion vectors output from each previous round are the initial motion vectors. Thus, starting with the initial motion vectors, the pixel information corresponding to the target point is predicted using the motion vectors and the reference frame. By determining the difference information, the reliability of the current motion vector can be determined, and further adjustments can be made to the corresponding motion vector based on the difference information:
[0075] Based on the differentiation information and the output motion vectors of each previous round, corresponding gradient information (i.e., gradient) is obtained. This gradient information describes the degree of influence of each previous round's output motion vector on the differentiation information. For example, embodiments of this application can determine the gradient of the prediction residual relative to each component of the current control point's motion vector. The components of a control point's motion vector include a lateral component and a vertical component. Thus, using the gradient information describing the degree of influence, the motion vector can be adjusted so that the adjusted motion vector can reduce the corresponding differentiation information. Embodiments of this application can update the current control point's motion vector based on the gradient and the prediction residual. Specifically, when adjusting using gradient information, it is necessary to adjust the output motion vectors of each previous round based on the digital information of the gradient information, with the goal of reducing differentiation information, thereby obtaining the corresponding current round's output motion vector.
[0076] In the video coding method provided in this application embodiment, when obtaining the motion vector corresponding to the target point in the prediction frame, multiple rounds of iterative optimization are performed on the initial motion vector to obtain a target motion vector that meets preset optimization conditions. Thus, after multiple iterations, the obtained target motion vector can describe the displacement of the target point between the prediction frame and the reference frame as accurately as possible, thereby reducing the amount of data containing distinguishing information between the two and improving the compression efficiency of video coding.
[0077] In each optimization round, the following steps are performed: Based on the reference frame and the output motion vectors from each previous round, the pixels of the current block are predicted. The difference between the original pixel values and the predicted results is then obtained. This difference information is used to measure the accuracy of the motion vector's description of the current block's displacement. Furthermore, gradient information is used to describe the degree of influence of the output motion vectors from each previous round on the difference information. This gradient information guides the adjustment of the motion vectors, ensuring that the adjusted motion vectors correspond to the lowest possible difference information, thereby improving the accuracy of the motion vector's description of the displacement.
[0078] Furthermore, in the process of adjusting motion vectors using gradient information, the digital information of the gradient information is used for related processing. The digital information of the gradient is quick to obtain and simple to process, which can quickly adjust the motion vectors and avoid processing methods that consume more resources. This improves the efficiency of video encoding and processing while reducing the computing resources consumed by video encoding and processing, thus effectively improving the efficiency and effect of video encoding and processing.
[0079] The application scenarios of the video encoding method provided in this application are described below.
[0080] Please refer to Figure 1C, which is a schematic diagram of an application scenario of the video encoding method provided in this application. This application scenario includes a server 101 and a client 102, which can communicate with each other. The communication method can be wired communication technology, such as communicating via a network cable or serial cable; or wireless communication technology, such as communicating via Bluetooth or Wireless Fidelity (WIFI), etc. No specific limitation is imposed.
[0081] Client 102 generally refers to devices that can send data requests, such as terminal devices, third-party applications accessible by terminal devices, or web pages accessible by terminal devices. Server 101 generally refers to devices that can send resources, such as terminal devices or servers.
[0082] Terminal devices include, but are not limited to, mobile phones, computers, smart medical devices, smart home appliances, vehicle terminals, or aircraft. Servers include, but are not limited to, cloud servers, local servers, or associated third-party servers. These can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Both client 101 and server 102 can use cloud computing to reduce the consumption of local computing resources; similarly, they can also use cloud storage to reduce the consumption of local storage resources.
[0083] As one embodiment, the server 101 and the client 102 can be the same device, or they can be different devices, or they can be different devices with some modules shared, etc., and there are no specific restrictions.
[0084] The video encoding method provided in this application embodiment will be described in detail below based on Figure 1C and the client. Please refer to Figure 2, which is a flowchart illustrating a video encoding method provided in this application embodiment.
[0085] S21, acquire the control point motion vectors of at least two control points of the current block in a video frame. Each control point motion vector represents the motion vector of the corresponding control point between the video frame and the reference frame. Here, the video frame is encoded using the reference frame, which can also be called a prediction frame. Control points are used for affine motion estimation. Control points can also be called target points. The number of control points is, for example, 2 or 3. In some embodiments, S21 may include: acquiring the initial motion vector of each target point in a prediction frame between a prediction frame and the reference frame, wherein a prediction frame is any one of the prediction frames associated with the reference frame. The initial motion vector is the initial control point motion vector.
[0086] Before introducing the process of obtaining the motion vector of the control points, it is necessary to first introduce the target point (control point):
[0087] As shown in Figure 3A, in the video encoding process, each video frame in the video frame sequence is typically encoded sequentially. When processing a single video frame, the video is divided into multiple coding units (CUs), and these CUs are encoded sequentially as well. Generally, the size of a coding unit in a video frame is variable, and the number of pixels it contains can be 64×64, 32×32, 16×16, 8×8, or different.
[0088] For a video frame being encoded (i.e., a predicted frame), the encoding process involves encoding multiple coding units in the video frame sequentially, and the coding unit being encoded can be called the current block (hereinafter referred to as the current block).
[0089] As discussed above, a crucial step in the video encoding process is determining the motion vector between two video frames to describe the motion of an object. Thus, by using only the pixel information of one video frame and combining it with this motion vector, the position information (i.e., the reference position) of a pixel in another video frame can be predicted through motion compensation. In this process, the video frame providing the pixel information is called the reference frame, while the video frame whose pixel position information has been predicted is called the prediction frame.
[0090] It should be noted that, as shown in Figure 3B, since video encoding is performed sequentially for each video frame, the video frame currently being encoded is the prediction frame mentioned above. The reference frame corresponding to this prediction frame can be any video frame with corresponding pixel information. For example, it could be an adjacent video frame that has already been encoded, or an encoded video frame that is separated from the prediction frame by multiple video frames. Correspondingly, for a video frame that serves as a reference frame, its associated prediction frame is not necessarily only the currently encoded video frame; it can also include multiple other video frames. The specific number can be determined according to actual needs, and this application does not impose any restrictions on this.
[0091] After clarifying the relationship between the predicted frame and the reference frame in the video to be processed, it is necessary to further explain the meaning of the motion vector between the predicted frame and the reference frame.
[0092] As the name suggests, motion vectors are vectors used to describe motion. In video encoding, motion vectors mainly refer to the displacement of an object between a reference frame and a prediction frame. Specifically, for each pixel, it means the displacement of the same pixel between the reference frame and the prediction frame.
[0093] However, in real life, the types of motion of objects are quite complex, including translation, rotation, scaling, stretching, and so on. Therefore, in the video coding process, in order to describe these types of motion using motion vectors, an affine motion estimation method is proposed. The following will introduce affine motion estimation using a coding unit in a video frame as an example:
[0094] Referring to Figure 3C, for the current block in a prediction frame undergoing video encoding, affine motion estimation estimates the motion of the entire current block using the motion vectors of pixels at fixed positions within that block. These fixed-position pixels are the target points mentioned in S21 above. In affine motion estimation, the target points are typically selected from the top-left corner (A), top-right corner (B), or bottom-left corner (C) of the current block, as shown in Figure 3C. Specifically, in a 4-parameter affine model, top-left corner (A) and top-right corner (B) are selected as target points (also known as control points). In a 6-parameter affine model, these three points—top-left corner (A), top-right corner (B), or bottom-left corner (C)—are selected as target points.
[0095] Taking a 4-parameter affine model as an example, after obtaining the motion vectors at points A and B, the motion vectors corresponding to these two points, along with the width and height of the current block, can be used to calculate the motion vectors of each pixel within the current block. However, in practical applications, due to the enormous computational cost of pixel-level affine mapping, the current block is usually divided into 16 sub-blocks of 4×4, as shown in Figure 4A. Then, the motion vector corresponding to the center pixel of each sub-block is calculated, and this motion vector describes the displacement of the entire sub-block.
[0096] As can be seen from the above process, the most important process for affine motion estimation of the current block is the acquisition of the motion vector of the target point. Therefore, the video coding method provided in this application embodiment will provide a detailed introduction to the acquisition of motion vectors.
[0097] The first thing to introduce is the affine model corresponding to the motion vector in affine motion estimation. Equations 1 and 2 below show the mathematical expressions for the 6-parameter affine model and the 4-parameter affine model, respectively.
[0098] Specifically, the expression for the 6-parameter affine model is: x′=ax+by+e
[0099] y′=cx+dy+f (Formula 1)
[0100] Where (x, y) are the pixel coordinates in the prediction frame currently being encoded, (x′, y′) are the corresponding pixel coordinates in the reference frame, parameters a, b, c, and d control the rotation, scaling, and cropping of the pixel coordinates, and parameters e and f control the translation of the pixel coordinates.
[0101] The 4-parameter affine model assumes that rotation and scaling are uniform, therefore its expression is: x′=ax-by+e
[0102] y′=bx+ay+f (Formula 2)
[0103] As can be seen from the above affine model, the motion vector of the target point (i.e. the motion vector of the control point) can be determined by the corresponding model parameters. In other words, once the corresponding model parameters are determined, the motion vector corresponding to each target point can be determined.
[0104] Therefore, having clarified the above introduction to the target points and motion vectors, the following will introduce the motion process corresponding to each target point.
[0105] S21 proposes that for each target point in a prediction frame, its corresponding initial motion vector needs to be obtained. The initial motion vector can be obtained in several different ways:
[0106] In one possible implementation, when obtaining the initial motion vector of the target points of the current block, selection can be based on the motion information of the encoded units surrounding the current block, or the corresponding encoded units in adjacent frames of the prediction frame containing the current block. Specifically, as shown in Figure 4A, since the pixels in the encoded units surrounding the current block in a prediction frame have spatial similarity to the current block, and the encoded units corresponding to the current block in adjacent frames have temporal continuity with the current block, the corresponding motion vectors can first be obtained from these other encoded units associated with the current block as a candidate motion vector set for the initial motion vectors of each target point of the current block.
[0107] After obtaining the candidate motion vector set, it is necessary to select the initial motion vector corresponding to each target point. This selection can be based on the Rate Distortion (RD) cost. The RD cost is a core indicator for evaluating the quality of a coding mode, used to balance the trade-off between the encoded bit rate and distortion. By calculating the RD cost, the candidate motion vector with the lowest RD cost can be selected from the set as the initial motion vector for each target point.
[0108] In some embodiments, step S21 may include S211 to S213.
[0109] S211: Construct a candidate list comprising at least one set of candidate motion vectors, where each set of candidate motion vectors is a control point motion vector of an encoded unit adjacent to the current block in the spatial or temporal domain. For example, a set of candidate motion vectors may consist of control point motion vectors of at least two control points of an encoded unit.
[0110] S212: Calculate the rate-distortion cost for each group of candidate motion vectors in the candidate list.
[0111] S213: The group of candidate motion vectors with the lowest rate-distortion cost in the candidate list is determined as the control point motion vectors of the at least two control points.
[0112] In summary, by constructing a candidate list containing motion vectors of control points of adjacent coded units in the spatial or temporal domains, and selecting the optimal set as the initial control point motion vectors for the current block based on rate-distortion cost, this technical solution provides a high-quality initial search point that achieves the best balance between coding cost and coding quality for subsequent iterative optimization processes. Constructing a candidate list using the correlation between spatial and temporal motion information is a standard-validated and efficient motion vector prediction technique. Based on this, the rate-distortion optimization criterion is introduced, and the Lagrange cost J = D + λ·R is calculated to jointly evaluate the number of coded bits and prediction distortion, enabling precise selection of the cost-effective initial motion vectors from multiple candidates. This allows iterative optimization to start from a position close to the global optimum, significantly reducing the required number of iterations and accelerating convergence. It improves coding efficiency while maintaining coding quality, representing an effective means of improving the performance of the entire affine motion estimation algorithm from the source.
[0113] In this embodiment, for each set of candidate control point motion vectors (CPMVs) in the candidate list, a complete affine motion compensation prediction is performed. Specifically, a weighted sum of the number of encoding bits required to encode using a set of CPMVs and the resulting image distortion is calculated. Finally, the set of CPMVs that minimizes this weighted sum (i.e., RD Cost) is selected as the optimal CPMV, and thus determined as the control point motion vectors for at least two control points of the current block.
[0114] This is an optimization process that seeks the best balance between "encoding cost" and "encoding quality".
[0115] The following are the specific calculation methods and examples:
[0116] The basic formula for calculating rate distortion cost J is: J = D + λ·R
[0117] Where: J (Cost): Rate-distortion cost, a comprehensive evaluation value. The smaller the value, the higher the cost-effectiveness of the mode.
[0118] D (Distortion): Distortion. Indicates the degree of pixel difference between the pixels of the predicted block and the original pixels of the current block after motion compensation using this set of CPMVs.
[0119] λ(Lambda): Lagrange multiplier. A weighting coefficient used to balance the proportion of bit rate and distortion in the total cost.
[0120] R (Rate): Bit rate. Represents the number of bits required to encode this set of CPMV (and related syntax elements).
[0121] The specific steps for calculating rate-distortion cost are as follows:
[0122] Assuming the current block (CU) uses a 4-parameter affine model, motion vectors need to be selected from the candidate list for two control points (top left and top right).
[0123] 1. Calculate the distortion D
[0124] Distortion measures how accurate the prediction is. For example, it can be calculated using the sum of absolute differences (SAD) or the sum of squared errors (SSE).
[0125] For example, a set of CPMVs can be extracted from the candidate list. Using the 4-parameter affine model formula, the motion vector of each pixel (or sub-block) within the current CU is derived from these two CPMVs. Based on the derived motion vectors, the pixel values at the corresponding positions are obtained from the reference frame to construct the prediction block for the entire current CU. The prediction block is then subtracted pixel by pixel from the original pixels of the current CU, and the absolute value (if using SAD) or square (if using SSE) is taken, and the sum is obtained to obtain the distortion value D. The smaller the SAD / SSE, the closer the motion described by the set of CPMVs is to the motion of the real object, and the closer the predicted image is to the original image.
[0126] 2. Calculate the bit rate R
[0127] Bit rate measures "how large this set of motion vector data is". The specific steps to calculate R are: calculate the motion vector difference (MVD). For each control point, MVD = candidate CPMV - predicted motion vector (MVP). MVP is usually derived from the motion information of adjacent coded blocks.
[0128] Furthermore, the horizontal and vertical components of the MVD are binarized. Based on the context model of entropy coding (such as CABAC), the number of bits R required to encode these MVDs (as well as syntax elements such as reference frame indices) is estimated.
[0129] After determining the distortion D and the bit rate R, the rate-distortion cost J can be determined.
[0130] Optionally, the candidate motion vector set may also include some preset motion vectors to address application scenarios where there are no coded adjacent blocks.
[0131] The above has introduced the relevant content of reference frames, prediction frames, target points, and motion vectors in video coding. After obtaining the initial motion vectors through the above methods, the following operations can be performed:
[0132] S22: Perform at least one round of iterative optimization on the control point motion vector. In some embodiments, the control point motion vector determined by step S21 may also be called the initial control point motion vector, or simply the initial motion vector.
[0133] In summary, by acquiring the control point motion vectors of at least two control points in the current block and performing at least one round of iterative optimization, and in each round of optimization, sequentially determining the prediction residual of the current block and the gradient of the prediction residual with respect to each component of the control point motion vector, and updating the control point motion vector based on the gradient and the prediction residual, this embodiment of the application can continuously correct the control point motion vector using the prediction residual and its gradient information, making the motion vector gradually approach the real motion state, thereby significantly reducing the prediction residual, improving the accuracy of motion compensation, and thus improving the compression efficiency of video coding. Simultaneously, this iterative optimization framework does not rely on a specific solution algorithm, providing a unified technical architecture for adopting efficient numerical calculation methods (such as binary bit-width elimination methods), thus laying the foundation for further reducing computational costs and increasing coding speed while ensuring coding quality.
[0134] In some embodiments, step S22 may include iteratively optimizing each of the obtained initial motion vectors until the obtained output motion vectors meet the preset optimization conditions.
[0135] After obtaining the initial motion vectors corresponding to each target point, it is necessary to optimize these initial motion vectors in multiple rounds until the final output motion vectors meet the preset optimization conditions.
[0136] The preset optimization condition can be set for either the number of optimization rounds or the output motion vector. For example, the preset optimization condition can be set to: the number of iterations does not exceed a preset number of rounds (e.g., 3 rounds). Thus, when the preset number of iterations is reached, the output motion vector obtained in this round is used as the target motion vector corresponding to the target point. Alternatively, the preset optimization condition can be set to: the RD cost corresponding to the output motion vector does not exceed a preset cost threshold. Thus, when the RD cost corresponding to the output motion vector in this round does not exceed the preset cost threshold, the output motion vector in this round can be used as the target motion vector corresponding to the target point, i.e., the updated control point motion vector finally obtained through iterative optimization.
[0137] The following will describe the optimization process for each round. Specifically, as shown in Figure 2, the following operations are performed in each round of optimization:
[0138] S221: Based on the reference frame and the current control point motion vector, determine the prediction residual of the current block. Here, the current control point motion vector is the motion vector obtained from the previous round of optimization, i.e., the output motion vector from the previous round. In other words, step S221 can predict the current block based on the reference frame and each output motion vector from the previous round, and determine the difference information (e.g., prediction residual) between the original pixel values of the current block and the prediction result of the current block. In some embodiments, the prediction residual of the current block may include the prediction residual of each pixel in the current block or the prediction residual of each sub-block of the current block. The prediction residual of each sub-block is, for example, the prediction residual of the center point of each sub-block.
[0139] In the first round of optimization, the initial motion vectors obtained need to be optimized. In subsequent rounds of optimization, the output motion vectors of the previous round need to be optimized. In other words, in the first round of optimization, the output motion vectors of the previous round are essentially the initial motion vectors.
[0140] Specifically, after the current round of optimization begins, it is necessary to first use the reference frame and the motion vectors output from each previous round to predict the current block and determine the difference information between the obtained prediction result and a prediction frame.
[0141] In some embodiments, step S221 may determine the reference block in the reference frame for the current block based on the current control point motion vector. Here, the current control point motion vector indicates the motion vector from the current block to the reference block. Based on this, the difference between the original pixel values of the current block and the reconstructed pixel values of the reference block can be determined as the prediction residual of the current block.
[0142] For example, as shown in Figure 4B, the pixel information (i.e., the reconstructed pixel value) of the reference position obtained based on the motion vectors can be derived using the output motion vectors of each previous round. Then, by using the pixel information in the prediction result and the original pixel value (i.e., the true pixel value) of the current block in the prediction frame, the difference information between the current block and the reference block can be obtained. This difference information includes multiple difference elements, each describing the prediction residual of a pixel.
[0143] In one possible implementation, in affine motion estimation, the motion vectors corresponding to each sub-block in the current block can be directly calculated using the motion vector of the target point and the width and height of the current block. Therefore, during the acquisition of the prediction result, the motion vector corresponding to each sub-block can be calculated using the previous round's output motion vector corresponding to the target point. Then, using the pixel information in the reference frame, all pixels in the current block are predicted, so that the prediction result includes the prediction information of each pixel in the current block. In this way, after obtaining the distinction information, the distinction elements included in the distinction information can describe the difference between each pixel in the current block and the prediction frame.
[0144] S222: Determine the gradient of the prediction residual with respect to each component of the current control point motion vector. For example, the partial derivatives of the prediction residual of the pixels in the current block with respect to each component of the current control point motion vector of the at least two control points are determined as the corresponding gradients. In other words, based on the differentiation information and the output motion vectors of each previous round, the corresponding gradient information (i.e., gradient) is obtained, wherein the gradient information is used to describe the degree of influence of each output motion vector of the previous round on the differentiation information. In this embodiment of the application, the gradient is determined by calculating the partial derivatives of the prediction residual of the pixels in the current block with respect to each component of the motion vector of at least two control points, which mathematically quantifies the sensitivity and direction of the influence of small changes in each component of the control point motion vector (e.g., translation, rotation, and scaling parameters of the affine model) on the prediction residual. Compared with approximation methods such as numerical difference, partial derivatives provide the most accurate first-order gradient information. Using this accurate gradient information to construct subsequent linear equation systems (including the calculation of autocorrelation matrix and cross-correlation vector) can ensure the correct direction and efficient convergence of the iterative optimization process, avoid slow convergence or getting trapped in local optima due to inaccurate gradient information, thus guaranteeing the final quality of motion estimation and providing an accurate and reliable raw data foundation for subsequent efficient elimination based on binary bit width.
[0145] For example, for a 4-parameter affine motion model, the current block includes two control points: the top-left control point and the top-right control point (x1, y1). The motion vector of the control point unique to the top-left control point is MV0(mv0). x mv0y Accordingly, the components of this motion vector include mv0. x and mv0 y The motion vector of the control point at the upper right is MV1(mv1). x mv1 y Accordingly, the components of this motion vector include mv1. x and mv1 y Taking the first and second pixels as examples, the prediction residual for the first pixel is R0, and the prediction residual for the second pixel is R1. Based on this, the partial derivatives (i.e., gradients) of R0 with respect to each component are respectively... The partial derivatives of R1 with respect to each component are as follows: In this way, the gradients can form a gradient matrix.
[0146] Therefore, after obtaining the differentiation information, in order to enable the differentiation information to guide the adjustment direction of the motion parameters, gradient information that describes the degree of influence of the motion vector on the differentiation information can be obtained.
[0147] In some embodiments, for gradient information, a common approach is to process the first derivative of each component of the output motion vector of each previous wheel based on the acquired differentiation information, and then use the result as the corresponding gradient information.
[0148] In one possible implementation, the gradient information can be obtained in the following way:
[0149] First, for each output motion vector of the previous wheel, perform the following operation: determine the vector parameters (i.e., components) corresponding to the output motion vector of the previous wheel.
[0150] As mentioned above, in the affine model, the motion vector corresponding to the target point is usually described by 4 or 6 parameters. The vector parameters corresponding to each target point are different. Therefore, in this process, it is necessary to determine the vector parameters corresponding to the output motion vector of the previous round.
[0151] Once the vector parameters and distinguishing elements are defined, the following operations can be performed on each vector parameter: based on the distinguishing elements corresponding to each pixel position, the gradient of a vector parameter is extracted to obtain the corresponding gradient elements, and the obtained gradient elements are used as gradient information.
[0152] The gradient extraction process is the specific process of the gradient information acquisition process mentioned above. It is applied to the distinguishing element and vector parameters. For a distinguishing element, the first derivative of each vector parameter is calculated, and the result is used as the corresponding gradient element.
[0153] After the above method is executed for each previous round's output motion vector and each vector parameter, the result is shown in Figure 4C. For gradient information, the gradient elements can be represented as a matrix. In this matrix, each element corresponds to a gradient element, which describes the influence of a vector parameter on a distinguishing element. Furthermore, in this matrix, each row corresponds to a pixel, and each column corresponds to a vector parameter. The only variable is the number of vector parameters and pixels. In the 4-parameter model, the matrix size can be N×4, while in the 6-parameter model, the matrix size can be N×6, where N is the number of pixels. Optionally, when the prediction result matches the prediction information corresponding to each pixel in the current block, each row of the matrix can correspond to a pixel, and the number of rows in the matrix is the number of pixels contained in the prediction information.
[0154] Thus, in this method, based on the distinguishing elements corresponding to each target point, gradient extraction is performed for each vector parameter. After obtaining the gradient elements that describe the degree of influence of the vector parameters on the distinguishing elements, all gradient elements can be used as the corresponding gradient information. The degree of influence of the motion vector on the distinguishing information described by this gradient information can guide the subsequent adjustment of the motion vector, thereby quickly obtaining the output motion vector that meets the preset optimization conditions.
[0155] S223: Update the motion vector of the current control point based on the gradient and the prediction residual.
[0156] In some embodiments, S223 includes: determining an adjustment amount for the current control point motion vector based on the gradient and the prediction residual; and updating the current control point motion vector based on the adjustment amount. By determining the adjustment amount based on the gradient and the prediction residual, the complex motion vector optimization problem can be transformed into a precise numerical calculation problem, determining the specific step size and direction for correcting each component. This allows each round of optimization to target and reduce the prediction residual, thereby accelerating the iteration convergence speed, approaching the optimal motion vector more quickly, and further improving coding efficiency.
[0157] In some embodiments, the present application embodiments may adjust the output motion vectors of each previous round based on the digital information of gradient information with the goal of reducing the distinguishing information, so as to obtain the corresponding output motion vectors of each current round.
[0158] In some embodiments, determining the adjustment amount for the motion vector of the current control point based on the gradient and the predicted residual includes:
[0159] A system of linear equations is constructed using the gradient and the prediction residuals. This system indicates the mapping from the gradient and the prediction residuals to the adjustment amount for the current control point's motion vector. In other words, the system of linear equations determines the adjustment amount for the current control point's motion vector. In some embodiments, the coefficient terms of this system of linear equations are aggregated from the autocorrelation of the gradient, characterizing the correlation between the changes in each CPMV component and the prediction residuals; the constant terms are aggregated from the cross-correlation of the gradient and the prediction residuals, characterizing the projection of the prediction residuals onto the change directions of each CPMV component. By solving this system of linear equations, the optimal CPMV adjustment amount that minimizes the updated prediction residuals can be obtained.
[0160] In some embodiments, the process of constructing the above-mentioned linear equation system involves first assembling the gradient and location weights into a Jacobian matrix, and then transforming it into a normal equation (i.e., the linear equation system mentioned above) using the least squares method. This process is described in detail below.
[0161] Assume there are K valid pixels (or sub-block center points) within the current block. For the i-th pixel:
[0162] Determine its horizontal gradient Gx at the corresponding position in the reference frame (i.e., the reference position). (i) and vertical gradient Gy (i) .
[0163] Determine its predicted residual R (i) (i.e., the difference between the original pixel value and the predicted pixel value obtained by motion compensation using the current CPMV).
[0164] The components of the current control point motion vector are P = (p1, p2, ..., pn), where n is the number of components (n = 4 for a 4-parameter affine model and n = 6 for a 6-parameter affine model).
[0165] The unknown to be solved is the increment of each component Δp = (Δp1, Δp2, ..., Δp) n ).
[0166] The following is the row-by-row construction process of the Jacobian matrix.
[0167] For each pixel i, calculate its prediction residual R.(i) The partial derivatives with respect to each CPMV component pj yield a row vector J of length n. i Where j takes values from 1 to n.
[0168] Among them, each partial derivative This indicates how much the prediction residual of pixel i will change when the component pj is finely adjusted by one unit under the current motion state. It is calculated as follows:
[0169] Where αj,x(i) and αj,y(i) are the spatial partial derivatives (i.e., position weights) of the j-th CPMV component with respect to the motion vector of the i-th pixel in the x and y directions, respectively. For example, for the horizontal component mv0 of the upper left control point in a 4-parameter affine model... x The position weight of the pixel at coordinates (x, y) is α = (1 - x / W), where W is the width of the current block. The horizontal gradient Gx (i) and vertical gradient Gy (i) These are the horizontal and vertical gradients of pixel i at its corresponding position (i.e., the reference position) in the reference frame, respectively.
[0170] Stacking the row vectors corresponding to all K pixels forms the Jacobian matrix J.
[0171] Since the Jacobian matrix J describes the sensitivity of the residuals to each component, and the increment Δp is the adjustment to the component itself, according to the first-order Taylor expansion, the change in the residuals can be approximately expressed as J·Δp. To make the updated residuals approach zero, the following system of equations should be satisfied:
[0172] J·Δp=-R
[0173] Where R = [R(1), R(2), ..., R(K)]T is the residual vector. The value of K is consistent with the number of prediction residuals in the current block.
[0174] Since the number of equations K in the system is usually much larger than the number of unknowns n, this overdetermined system of equations has no exact solution. Therefore, it is transformed into normal equations using the least squares method:
[0175] (J T J)·Δp=-J T R
[0176] Let the coefficient matrix H = J T J (of size n×n), constant term vector b = -J T R (of size n×1) can be simplified to the normal equation as follows:
[0177] H·Δp=b
[0178] Where: the coefficient matrix H is the autocorrelation aggregation of the gradient (i.e., the autocorrelation matrix), and its p-th row and q-th column elements are... It aggregates the information of the "gradient × spatial partial derivative" product pairs of all pixels in the current block, representing the correlation established by the gradient between the changes of two different CPMV components.
[0179] The constant term vector b is the cross-correlation aggregation (i.e., cross-correlation matrix) of the gradient and the predicted residual. Its p-th element... This augmented matrix aggregates the information from the product of "gradient × spatial partial derivative × residual" for all pixels in the current block, representing the projection of the current prediction residual onto the directions of change of each CPMV component, i.e., the optimal adjustment direction indicated by the prediction residual. In this augmented matrix, each row of the coefficient matrix H corresponds to a component adjustment to be solved (i.e., a normal equation), and each column corresponds to the coefficient of each component adjustment in the normal equation. Each row of the constant term vector (i.e., the constant column) b corresponds to the weighted sum of residuals corresponding to that component adjustment.
[0180] After determining the linear equation system, embodiments of this application can eliminate variables from the linear equation system based on the binary bit width of the elements in the augmented matrix of the linear equation system to obtain an eliminated linear equation system. The augmented matrix of the eliminated linear equation system is a row echelon matrix. The augmented matrix of the linear equation system may include a coefficient matrix (e.g., H in the above text) and a constant column (e.g., the constant term vector b in the above text). Each column in the coefficient matrix corresponds to a component (i.e., a component of the control point motion vector).
[0181] The embodiments of this application can determine the adjustment amount of the motion vector of the current control point based on the eliminated linear equations.
[0182] In summary, the embodiments of this application can further model the iterative optimization problem of motion vectors as a problem of constructing and solving a system of linear equations, making the mapping relationship between the optimization objective and the solution method clear and explicit. Its core technical effect lies in the fact that, in the crucial step of solving this system of linear equations—which determines the computational cost of the encoder—it abandons the computationally intensive and time-consuming division operation in the traditional Gaussian elimination method. Instead, it performs elimination based on the binary bit width of the elements in the augmented matrix, ultimately obtaining a system of eliminated equations in the form of a row echelon matrix. This approach allows the hardware to replace complex division operations with simpler and faster shift operations when performing elimination, significantly reducing the computational cost per iteration and hardware implementation latency, thus achieving a substantial improvement in video encoding rate from the underlying computational architecture.
[0183] In some embodiments, the linear equation system is eliminated based on the binary bit width of the elements in the augmented matrix of the linear equation system to obtain an eliminated linear equation system, including:
[0184] Perform multiple rounds of elimination on the augmented matrix; each round of elimination includes:
[0185] The first element to be selected as the pivot is chosen from the current elimination column. Here, the elimination column refers to a column in the augmented matrix that is being eliminated in this round. More specifically, an elimination column is a column of the coefficient matrix (i.e., the coefficient part) contained in the augmented matrix. After this round of elimination, all elements (i.e., coefficients) in the elimination column except the pivot become zero, or the elements below the pivot become zero. For example, for the current elimination column, a non-zero coefficient can be selected as the pivot. Another example is the element located on the diagonal of the elimination column. The element with the largest non-zero absolute value in the current elimination column can be selected as the pivot.
[0186] Calculate the first product of the first element and the second element, where the second element is any element in the augmented matrix except for the row containing the first element. In some embodiments, the second element may be referred to as the element to be updated, and its selection range is: in every other row except the row containing the pivot, all elements located in the current elimination column and to its right (including elements in the coefficient matrix and constant column). In other words, the second element is the element in the augmented matrix except for the row containing the first element, located in the current elimination column and to its right.
[0187] Calculate the second product of the third element and the fourth element. The third element is in the same column as the first element and in the same row as the second element, and the fourth element is in the same row as the first element and in the same column as the second element. Update the second element based on the first product and the second product, including one of the following:
[0188] Calculate the difference between the first product and the second product; perform a first right shift operation on the difference to obtain an updated second element, wherein the right shift bit is equal to the binary bit width of the first element; and
[0189] Perform a first right shift operation on the first product to obtain a first result; perform a first right shift operation on the second product to obtain a second result; and use the difference between the first result and the second result as the updated second element.
[0190] In summary, this embodiment precisely and equivalently replaces the core division operation in the elimination process with a composite operation of "multiplication-difference-shift" or "shifting separately and then taking the difference," where the number of bits shifted in the right shift operation is determined by the binary bit width of the pivot (first element). Replacing division with a right shift operation avoids the large logic gate delays and resource consumption of traditional dividers, enabling the elimination step to be completed within a shorter clock cycle and naturally suitable for pipelined implementation in hardware. Furthermore, this elimination operation performs only localized calculations on each element in the augmented matrix, eliminating the serial dependency between different rows in traditional Gaussian elimination, further improving instruction-level parallelism. This not only reduces the computational hardware cost and power consumption of the video encoding process but also significantly improves hardware processing throughput, representing a key technological innovation for solving the hardware timing impact problem of traditional solutions and ensuring the real-time performance of video encoding.
[0191] In some embodiments, each round of elimination further includes:
[0192] The sum of the binary bit width of the first element and the binary bit width of the second element is used as the first estimated bit width.
[0193] The sum of the binary bit width of the third element and the binary bit width of the fourth element is used as the second estimated bit width;
[0194] Determine whether one of the first estimated bit width and the second estimated bit width exceeds a preset bit width threshold;
[0195] When it is determined that either the first estimated bit width or the second estimated bit width exceeds a preset bit width threshold, a second right shift operation is performed on the element with the larger binary bit width between the first and second elements; a second right shift operation is also performed on the element with the larger binary bit width between the third and fourth elements; the right shift number of the second right shift operation is the difference between the binary bit width of the first element and the preset bit width threshold. Based on this, the process of calculating the first product, the second product, and determining the first and second results is further executed.
[0196] In addition, when it is determined that the first estimated bit width and the second estimated bit width do not exceed the preset bit width threshold, the process of calculating the first product, the second product, and determining the first result and the second result described above is directly executed.
[0197] In summary, this embodiment of the application, by estimating the sum of the bit widths of each element participating in the multiplication operation before performing the binary bit-width-based elimination operation and comparing it with a preset bit-width threshold, and performing a second right-shift operation for pre-scaling processing on elements with larger bit widths, effectively prevents the problem of the result bit width doubling due to the multiplication of two binary numbers, exceeding the hardware preset data bit width. This scheme ensures that all intermediate and final results can be correctly represented and stored throughout the entire "multiplication-shift" elimination process, guaranteeing the robustness and numerical accuracy of the calculation results. Simultaneously, by selecting elements with larger bit widths for pre-scaling, the calculation precision is preserved to the greatest extent, which is an effective strategy that balances calculation correctness and accuracy while eliminating division operations, significantly enhancing the stability and reliability of the algorithm in hardware implementation in video coding.
[0198] In some embodiments, the coefficient portion of the augmented matrix of the eliminated linear equation system is a diagonal matrix;
[0199] The determination of the adjustment amount for the motion vector of the current control point based on the eliminated linear equation system includes: calculating the adjustment amount of each component in parallel based on the diagonal matrix.
[0200] In summary, the embodiments of this application can further eliminate the coefficients of the eliminated linear equation system into a diagonal matrix, thus completely decoupling the original multivariate simultaneous equation system into multiple independent linear equations in one variable. Based on this diagonal matrix structure, the adjustment amount of each control point motion vector component can be calculated independently and simultaneously, completely breaking the dependency chain that traditional row echelon matrices require strict serial back-substitution. This scheme greatly improves the parallelism of solving the adjustment amount, a key step in iterative optimization, and can fully utilize the parallel computing capabilities of modern multi-core processors or dedicated video coding hardware, significantly reducing the computation time of each round of iterative optimization, thereby significantly improving video coding efficiency.
[0201] In some embodiments, after obtaining the gradient information and discrimination information corresponding to the previous round's output motion vector, a four- or six-variable system of equations (i.e., the linear system of equations mentioned above) can be obtained based on the gradient information and discrimination information. As shown in Figure 5A, each unknown in this system of equations is an adjustment amount for a vector parameter (i.e., a component of the previous round's output motion vector). The coefficient matrix (including coefficient elements) corresponding to the system of equations is the autocorrelation matrix of the gradient mentioned above, and the constant column (including constant column elements) in the system of equations is the cross-correlation matrix between the gradient and the prediction residual. After solving this system of equations, each vector parameter can be assigned an adjustment amount, which can then serve as the data basis for subsequent adjustments to the motion vector.
[0202] In solving systems of equations, Gaussian elimination is a commonly used method. The core idea of Gaussian elimination is to transform the system of equations into a row echelon form matrix through row operations, and then substitute back to solve. Specifically, for the augmented matrix corresponding to a system of equations, as shown in Figure 5B, after row operations, the matrix is transformed into a matrix similar to a row echelon form, where the elements in the lower left corner of the diagonal of the coefficient matrix are reduced to 0. Then, by substituting the solution for the unknowns in the last row back into the row echelon form matrix, the solutions for the remaining unknowns can be solved sequentially.
[0203] When performing elimination, the formula used to update the element in the i-th row is:
[0204] Where a represents a coefficient, b represents a constant, and i > k.
[0205] However, in this process, when eliminating elements from the augmented matrix, a large number of division operations are required, as shown in Equation 3: that is, in Equation 3... This division calculation.
[0206] In the field of computer science, division calculations require a large amount of computing resources. On the other hand, the amount of data involved in division operations is enormous, and each division operation takes a long time. This puts a great strain on the timing constraints of the hardware itself, making it difficult for hardware devices to quickly and accurately complete the optimization iteration of motion vectors for target points under such circumstances.
[0207] Therefore, the video coding method proposed in this application suggests that the digital information of the coefficient matrix of the linear equation system is directly used to adjust each of the above-mentioned output motion vectors to obtain the corresponding current-round output motion vector.
[0208] In one possible implementation, to obtain the output motion vector for this round, the following operations can be performed:
[0209] First, determine the multiple elements included in the coefficient matrix.
[0210] Next, based on the digit information of each principal element selected from the coefficient matrix, the digits of each other element in the augmented matrix are adjusted, and based on the adjusted coefficient matrix and the distinguishing information, the parameter adjustment information (i.e., the adjustment amount) of each previous round output motion vector is obtained.
[0211] After constructing the system of equations using gradient and distinguishing information, each element in the coefficient matrix is a corresponding coefficient element in the system of equations. At this point, the coefficients need to be processed to reduce the difficulty of solving the system of equations.
[0212] Since the proposed solution is a video coding technology applied in computer equipment, in order to improve video coding efficiency, this application proposes to abandon the division operation in the process of solving equations, and instead use digital processing methods in the field of computer technology to predict the result of each division operation.
[0213] Therefore, this process proposes using the bit information (i.e., binary bit width) of the selected pivot element to adjust the bit width of other elements in the augmented matrix. Here, the selected element can also be called the pivot element, i.e., the first element.
[0214] In one possible implementation, the process of adjusting the digits of other elements in the augmented matrix using the digit information of the selected elements can be performed in the following manner:
[0215] For each of the other elements, perform the following operations:
[0216] First, based on the preset correspondence, select an element corresponding to another element from the selected principal elements as the target principal element (i.e., the current first element).
[0217] The preset correspondence can be described as shown in Figure 5C. As shown in Figure 5C, assuming that it describes a coefficient matrix to be processed (the number of elements in the coefficient matrix is only for illustrative purposes and does not represent the number of elements in actual applications), then the selected coefficients are usually the pivot elements of the coefficient matrix, that is, the elements in the diagonal of the coefficient matrix.
[0218] For example, as shown in Figure 5C, Gaussian elimination obtains the result by sequentially eliminating coefficients in each column of the matrix. Therefore, each elimination process requires a complete elimination of the column containing a vector parameter. This is reflected in the selection of the target pivot: when eliminating vector parameters in the first column, the pivot of the first column is selected as the target pivot to process the other elements. Similarly, when eliminating vector parameters in the second column, the pivot of the second column is selected as the target pivot to process the other elements.
[0219] Second, after selecting the target pivot, the other elements to be processed, as well as the target pivot, can be converted into a preset number system type, thereby obtaining the converted target pivot and one of the other elements. The preset number system type can be a binary number system.
[0220] Finally, based on the digit information of the transformed target principal element, and in conjunction with the target principal element, the digits of another element after the transformation can be adjusted.
[0221] For example, taking the matrix shown in Figure 5D as an example, and combining it with the content of Formula 3 above, the above content will be explained. As shown in Figure 5D, assuming that in the matrix shown in Figure 5D, element D (i.e., the second element) is the element that needs to be adjusted during the current elimination process, element M (i.e., the first element) is the target pivot corresponding to element D, and element L (i.e., the third element) is the element represented in Formula 3. Element C (i.e., the fourth element) is represented in Formula 3. In other words, the row containing element L is the same as the row containing element D, the column containing element L is the same as the column containing element M, and the row containing element C is the same as the row containing element M, and the column containing element C is the same as the column containing element D. Therefore, for element D, the relationship before and after adjustment is as shown in Formula 4 below:
[0222] D 新 =D 旧 -(L÷M)·C (Formula 4)
[0223] In other words, the content of Formula 4 can be expressed as Formula 5, that is:
[0224] D 新 = (M·D 旧 -L·C)>>log2M (Formula 5)
[0225] In Formula 5, >> means shifting to the right, and log2M represents the bit information of element M in binary.
[0226] Furthermore, the digital adjustment processing for other elements will be illustrated below with examples from Figure 5D and Figure 5E.
[0227] For example, as shown in Figure 5E, for the processing method shown in Formula 5, assuming that a digit adjustment is needed for element D as shown in Figure 5D, then for elements M, L, and C related to element D, corresponding digit adjustment is required simultaneously. That is, after multiplying element M and element D, the product is reduced in number of bits, i.e., shifted to the right. The number of bits shifted is determined by the binary digit information of element M. For example, assuming the value of element M is 8, its binary digit information is 3. Then, the number of bits shifted to the right after multiplying element M and element D is the digit information of element M, i.e., 3.
[0228] Similarly, the same process can be applied to the product of element L and element C. After obtaining the result of the shifting process, the difference between the two is the adjusted value of element D.
[0229] Thus, by using the above processing method, the division operation in Formula 4 is transformed into digital processing in computer technology. While meeting the data precision requirements in the field of video encoding, it avoids the occupation of a large amount of complex computing resources in the division operation. Instead, only shift processing is required. Furthermore, since the main body of this digital processing only requires relatively simple bit processing operations, it can significantly improve the acquisition process of the current output motion vector, thereby increasing the processing speed of video encoding.
[0230] In one possible implementation, as shown in Formula 5, during the above-mentioned digit adjustment process, there may be a situation where two elements are multiplied. In order to avoid the two elements being multiplied directly, which would cause the number of digits in the result to overflow and result in an inaccurate result, this application proposes the following method to ensure the accuracy and stability of the digit adjustment process.
[0231] Specifically, the first step is to fuse the digit information of the transformed target principal component and the digit information of another transformed element. For example, the fusion process involves summing the digit information of the transformed principal component with the digit information of the other transformed element.
[0232] Thus, when the fused digital information does not exceed the preset bit width threshold, the digital adjustment of another element after conversion can be directly performed in the manner shown in Formula 5.
[0233] When the fused digital information exceeds the preset bit width threshold, it is necessary to perform bit reduction processing on the larger value of the converted principal element's digital information and the converted digital information of another element, based on the first difference between the fused digital information and the preset bit width threshold.
[0234] In this process, because the larger value in the digit information is selected for bit reduction, the precision issues caused by bit reduction can be minimized in both elements. For example, the precision issues caused by reducing a 10-bit binary number to a 9-bit binary number are much smaller than the precision issues caused by reducing a 3-bit binary number to a 2-bit binary number.
[0235] Furthermore, when the target principal element after digit reduction is obtained, based on the second difference between the digit information of the target principal element and the first difference, the product between another element and the target principal element after digit reduction is processed to reduce the number of digits.
[0236] When another element after number reduction is obtained, based on the second difference between the number information of the target pivot and the first difference, the product between the target pivot and the other element after number reduction is processed to reduce the number of bits.
[0237] Since the number of bits to be reduced in Formula 5 is the digit information of the target pivot, and in the previous process, the first difference digit reduction has already been performed on the digit information of the transformed target pivot, or the digit information of another transformed element. Therefore, after completing the digit reduction based on the first difference, the resulting product needs to undergo a second digit reduction based on the difference between the digit information of the target pivot and the first difference.
[0238] For example, as shown in Figure 5F, combined with the matrix information shown in Figure 5D, if it is necessary to adjust element D, it is first necessary to judge the multiplied elements M and D. When the fusion value of the digital information of the two exceeds the set threshold, it can be determined that the result of multiplying the two will exceed the value that the computer can currently represent. Therefore, it is necessary to perform bit reduction processing on the two in advance.
[0239] To ensure the accuracy of the results, it is necessary to process the element with larger digit information in elements M and D to reduce the error rate.
[0240] Thus, after the first stage of bit reduction processing, the two elements can be multiplied, and the resulting product can be subjected to a second stage of bit reduction processing. In the second stage of bit reduction processing, the number of bits shifted to the right only needs to be the second difference between the digit information of element M and the first difference.
[0241] In this way, the reduction of the number of bits, which was originally done all at once, is divided into two stages. This ensures the accuracy of the result after the reduction of the number of bits and also prevents the result of the element-wise multiplication from exceeding the data range that the computer can express, thus ensuring the reliability of the implementation of the solution.
[0242] Similarly, the product of element L and element C in Formula 5 can be processed in the same way as described above, and then another element after the bit adjustment can be obtained through Formula 5.
[0243] In the above method, the two-stage bit reduction process ensures that bit overflow will not occur when multiplying two elements, thus preventing inaccurate data recording and improving the accuracy and reliability of the solution. Furthermore, in the first stage of bit reduction, the element with the larger bit count is selected for processing, which also ensures that the accuracy error during bit reduction is small, avoiding impact on the subsequent acquisition of motion vectors.
[0244] The above describes the processing of elements in the coefficient matrix. In the process of solving equations, in addition to processing the coefficient elements, the elements in the constant column also need to be processed.
[0245] In one possible implementation, the following processing also needs to be performed on the constant column:
[0246] Based on the digit information of each selected principal element, the digits of each element in the constant column are adjusted.
[0247] Therefore, in addition to performing the aforementioned digit adjustment process on the coefficient elements, the same digit adjustment process also needs to be performed on the elements in the constant column. It should be noted that this digit adjustment process can refer to the digit adjustment process performed on the elements, and its specific execution principle is based on the processing method reflected in Formula 5, which will not be repeated here.
[0248] After adjusting the constant column in the above manner, based on the adjusted coefficient matrix and the adjusted constant column, the corresponding parameter adjustment element (i.e. the adjustment amount for each component) is obtained for each vector parameter, and the obtained parameter adjustment element is used as parameter adjustment information.
[0249] Thus, for the augmented matrix corresponding to the system of equations, after performing elimination on each column of the coefficient matrix, the corresponding row echelon form matrix can be obtained. In other words, the coefficient matrix of the system of equations is an upper triangular matrix.
[0250] At this point, by substituting the solution obtained through the last vector parameter back into the other equations, we can obtain the solution corresponding to each vector parameter.
[0251] It should be noted that, since only a finite number of division operations are required in this process, the division operation can be performed directly during the back substitution solution process, or the digit information of the divisor can be used for digit adjustment. This application does not impose any restrictions on this.
[0252] Thus, after obtaining the parameter adjustment information based on the coefficient matrix and constant column, the parameter adjustment information can be used to reduce the prediction residual. The following steps are performed on each previous round output motion vector: each vector parameter of a previous round output motion vector is adjusted to obtain the corresponding current round output motion vector.
[0253] In this method, the digit information of the principal element in the coefficient matrix is used to adjust the digits of other elements. Through simple shifting, the resource-intensive division operation is replaced. While ensuring controllable computational accuracy, the computational efficiency of video encoding is greatly improved, and the computational resources required for the video encoding process are reduced.
[0254] The above describes the digit adjustment process for coefficient elements (elements of the coefficient matrix) and constant column elements during parameter adjustment. To further improve video encoding efficiency, the following methods can also be used during parameter adjustment information acquisition:
[0255] In one possible embodiment, when adjusting the coefficient elements, the following operations may also be performed:
[0256] As shown in Figure 6A, in the aforementioned adjustment process of the coefficient elements, the original augmented matrix is converted into a corresponding row echelon form matrix. Thus, after obtaining the corresponding system of equations, solving for the unknowns requires sequentially substituting back into the equations to obtain the solutions for the remaining unknowns. This process is overly dependent, and the entire back-substitution process is a serial process for the executing entity, which can significantly impact the hardware timing of the executing entity. Therefore, to improve the parallel processing efficiency of the back-substitution process, the embodiments of this application propose the following implementation method:
[0257] For each vector parameter, perform the following operations sequentially:
[0258] First, from the multiple previous adjustment elements corresponding to a vector parameter, select the previous adjustment element whose value matches the preset filtering conditions, and use it as the hit element (i.e., the principal element); when adjusting the first vector parameter, the previous adjustment element is the element that has not been adjusted.
[0259] It's important to clarify that the elimination process is performed sequentially on each column. Therefore, when eliminating a column, you first need to select the previous adjustment element whose value matches the preset filter criteria, i.e., select the pivot element. This filter criteria can be: the previous adjustment element with the largest non-zero value, or simply any previous adjustment element that is non-zero.
[0260] Next, based on the digit information of the pivot, the following operations are performed on each of the other previous round adjustment coefficient elements among the multiple previous round adjustment coefficient elements: the value of a previous round adjustment element is adjusted to a preset value; the hit row corresponding to a previous round adjustment element is determined, and based on the digit information of the pivot, the digits of the other elements are adjusted to obtain the corresponding current round adjustment element.
[0261] As shown in Figure 6B, in the process of eliminating a vector parameter, assuming that the vector parameter is located in the second column, the hit element is the element in the second row of the second column of the matrix. Then, for each other adjustment element in this column from the previous round, the following operations must be performed:
[0262] Adjust the value of one element in the second column (excluding the pivot) shown in Figure 6B to a preset value. Optionally, this preset value can be zero.
[0263] In this way, we can continue to adjust the digits of other elements in the row where the principal element is located to obtain the corresponding adjustment elements for this round. In the matrix shown in Figure 6B, this is to adjust the digits of other adjustment coefficient elements in the row where the adjusted element of the previous round is located.
[0264] Thus, in this method, for each vector parameter, during the elimination stage, all elements except the pivot can be eliminated. After the elimination of all vector parameters is completed, a set of equations that are independent of each vector parameter can be obtained. This avoids the serial back substitution operation required for a set of equations with multiple unknowns, thereby realizing parallel processing of unknowns, improving the efficiency of obtaining parameter adjustment information, and thus improving the efficiency of video coding.
[0265] Furthermore, after adjusting the digits of the other coefficient elements corresponding to a hit row each time, the following operations can be performed:
[0266] Specifically, for each of the other adjustment coefficient elements from the previous round, perform the following operations:
[0267] In the previous round adjustment constant column, determine a previous round adjustment constant column element corresponding to another previous round adjustment coefficient element; when adjusting for the first vector parameter, the previous round adjustment constant column is the constant column that has not been adjusted; each constant column element contained in the constant column corresponds to a row.
[0268] As shown in Figure 6C, when adjusting other previous round adjustment elements using the hit element corresponding to a vector parameter, it is also necessary to adjust the previous round adjustment constant column element corresponding to each other previous round adjustment element. That is, based on the digit information of the principal element, the digit of a previous round adjustment constant column element is adjusted to obtain the corresponding current round adjustment constant column element.
[0269] Thus, after each of the other previous adjustment elements has been adjusted, the current adjustment constant column can be obtained by matching the previous constant column element corresponding to the obtained current adjustment constant column element.
[0270] In this method, while processing other adjustment coefficient elements from the previous round, the corresponding constant column elements are also adjusted, which improves the accuracy of obtaining parameter adjustment information.
[0271] In one possible implementation, after adjusting the output motion vectors of each previous wheel using the above method, an RD cost calculation can be performed on the adjusted output motion vectors of each previous wheel. If the RD cost of the adjusted output motion vectors of each previous wheel is determined to be less than that of the unadjusted output motion vectors of each previous wheel, then the adjusted output motion vectors of each previous wheel are used as the output motion vectors of each current wheel. Conversely, if the RD cost of the adjusted output motion vectors of each previous wheel is not less than that of the unadjusted output motion vectors of each previous wheel, then the output motion vectors of each previous wheel can be directly used as the output motion vectors of each current wheel.
[0272] In this way, it can be ensured that after each round of optimization, the RD cost corresponding to the output motion vector can generally show a downward trend, so as to quickly obtain the output motion vector that meets the optimization conditions.
[0273] The above describes the video coding method and its various possible implementations provided in this application. In the video coding method provided in this application, when obtaining the motion vector corresponding to the target point in the prediction frame, multiple rounds of iterative optimization are performed on the initial motion vector to obtain a target motion vector that meets preset optimization conditions. Thus, after multiple iterations, the obtained target motion vector accurately describes the displacement of the target point between the prediction frame and the reference frame, thereby reducing the amount of prediction residual data between the two and improving the compression efficiency of video coding.
[0274] In each optimization round, the following steps are performed: Based on the reference frame and the output motion vectors from each previous round, the pixel information of each target point is predicted. The difference between the predicted result and the predicted frame is then obtained. This difference is used to measure the accuracy of the motion vector's description of the target point's displacement. Furthermore, gradient information is used to describe the degree of influence of each previous round's output motion vector on the difference. Thus, the gradient information guides the adjustment of the motion vector, ensuring that the adjusted motion vector corresponds to the lowest possible difference, thereby improving the accuracy of the motion vector's description of the displacement.
[0275] Furthermore, in the process of adjusting motion vectors using gradient information, the digital information of the gradient information is used for related processing. The digital information of the gradient is quick to obtain and simple to process, which can quickly adjust the motion vectors and avoid processing methods that consume more resources. This improves the efficiency of video encoding and processing while reducing the computing resources consumed by video encoding and processing, thus effectively improving the efficiency and effect of video encoding and processing.
[0276] The video encoding method and its various possible implementations provided in the embodiments of this application have been described above. In practical applications, the above-described implementations can be combined in different ways. The process will be described below through a possible combined implementation.
[0277] As shown in Figure 7, in affine estimation for video coding, for the current block undergoing coding, the first step is to obtain the initial motion vector corresponding to the target point. This initial motion vector is mainly obtained from other coded units adjacent to the current block. Specifically, motion vectors at other points corresponding to the target point can be obtained from spatially and temporally adjacent coded units as a candidate motion vector set. Then, from the candidate motion vector set, the set of motion vectors with the minimum RD cost is selected as the corresponding initial motion vector.
[0278] Thus, after obtaining the initial motion vector, a multi-round iterative optimization process can begin based on that initial motion vector. Specifically, the following operations can be performed during each iteration:
[0279] By inputting the motion vectors output from each previous round into the current round for adjustment processing, as well as the pixel information in the reference frame, the pixel information of the target point in the prediction frame and the pixel information of the current block where the target point is located can be predicted. After obtaining the corresponding prediction results, the prediction results are compared with the prediction frame to obtain the difference information composed of the difference elements corresponding to each pixel in the current block.
[0280] Furthermore, by utilizing the distinguishing information and the vector parameters in the motion vector, the corresponding gradient information is obtained, so that each coefficient element in the gradient information represents the degree of influence of a vector parameter on the distinguishing element corresponding to a pixel.
[0281] Thus, by solving a system of equations based on gradient information and differentiation information, the corresponding parameter adjustment information is obtained. Based on this parameter adjustment information, the output motion vectors of each previous round are adjusted to obtain the output motion vector of the current round.
[0282] In solving the system of equations, the number of digits of the selected principal components is used to reduce the number of digits of other coefficient elements, thereby adjusting the augmented matrix corresponding to the system of equations. This makes the augmented matrix in the form shown in Figure 7. In the adjusted augmented matrix, there is only one unknown in each equation. Therefore, the back substitution process can be processed in parallel, so as to quickly obtain the solution corresponding to each unknown, that is, to quickly obtain the corresponding parameter adjustment information.
[0283] Based on the same inventive concept, this application provides a video encoding apparatus capable of implementing the functions corresponding to the aforementioned video encoding method. Referring to Figure 8, the apparatus includes an acquisition module 801 and a processing module 803, wherein:
[0284] In some embodiments, the acquisition module 801 is used to acquire control point motion vectors of at least two control points of the current block in a video frame; each control point motion vector represents the motion vector of the corresponding control point between the video frame and the reference frame;
[0285] Processing module 803 is used to perform at least one round of iterative optimization on the motion vector of the control point; wherein each round of iterative optimization includes:
[0286] Based on the reference frame and the motion vector of the current control point, determine the prediction residual of the current block;
[0287] Determine the gradient of the predicted residual with respect to each component of the motion vector of the current control point;
[0288] The motion vector of the current control point is updated based on the gradient and the prediction residual.
[0289] The more specific implementation of the video encoding device is the same as the video encoding method described above, and will not be repeated here.
[0290] Please refer to Figure 9, which illustrates a computer device 900 provided in this embodiment. This computer device 900 can be, for example, the client 91 or server 92 shown in Figure 1C. The current and historical versions of the data storage program, as well as the corresponding application software, can be installed on the computer device 900. The computer device 900 includes a processor 980 and a memory 920. In some embodiments, the computer device 900 may include a display unit 940, which includes a display panel 941 for displaying a user-interactive interface, etc.
[0291] In one possible embodiment, the display panel 941 may be configured in the form of a liquid crystal display (LCD) or an organic light-emitting diode (OLED).
[0292] The processor 980 is used to read a computer program and then execute the methods defined by the computer program. For example, the processor 980 reads a data storage program or file, thereby running the data storage program on the computer device 900 and displaying the corresponding interface on the display unit 940. The processor 980 may include one or more general-purpose processors, and may also include one or more DSPs (Digital Signal Processors) for performing related operations to implement the technical solutions provided in the embodiments of this application.
[0293] The memory 920 generally includes main memory and secondary storage. Main memory can be random access memory (RAM), read-only memory (ROM), and cache, etc. Secondary storage can be a hard disk, optical disk, USB flash drive, floppy disk, or magnetic tape drive, etc. The memory 920 is used to store computer programs and other data. The computer programs include applications corresponding to each client, and other data may include data generated after the operating system or applications are run, including system data (e.g., operating system configuration parameters) and user data. In this embodiment, the computer program is stored in the memory 920, and the processor 980 executes the computer program in the memory 920 to implement any of the methods described in the preceding figures.
[0294] The aforementioned display unit 940 is used to receive input digital information, character information, or contact touch operations / non-contact gestures, and to generate signal inputs related to user settings and function control of the computer device 900. Specifically, in this embodiment, the display unit 940 may include a display panel 941. The display panel 941, for example, is a touch screen, which can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or on the display panel 941), and drive corresponding connection devices according to a pre-set program.
[0295] In one possible embodiment, the display panel 941 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller. The touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 980. It can also receive and execute commands from the processor 980.
[0296] The display panel 941 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the display unit 940, in some embodiments, the computer device 900 may also include an input unit 930. The input unit 930 may include an image input device 931 and other input devices 932, wherein the other input devices may include, but are not limited to, one or more of the following: a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick.
[0297] In addition to the above, the computer device 900 may also include a power supply 990 for powering other modules, an audio circuit 960, a near-field communication module 970, and an RF circuit 910. The computer device 900 may also include one or more sensors 950, such as an accelerometer, a light sensor, and a pressure sensor. The audio circuit 960 specifically includes a speaker 961 and a microphone 962, for example, the computer device 900 can use the microphone 962 to collect the user's voice and perform corresponding operations.
[0298] As one embodiment, the number of processors 980 can be one or more, and the processors 980 and the memory 920 can be coupled together or relatively independent.
[0299] As one embodiment, the processor 980 in FIG9 can be used to implement the functions of the acquisition module and the processing module shown in FIG9.
[0300] As one embodiment, the processor 980 in Figure 9 can be used to implement the functions corresponding to the server or terminal device discussed above.
[0301] In some possible implementations, various aspects of the video coding method provided in this application may also be implemented in the form of a computer program product, which includes a computer program that, when run on an electronic device, causes the electronic device to perform the steps of the video coding method according to the various exemplary embodiments of this application described above.
[0302] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0303] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of software products, for example, through computer program products. These computer program products are stored in a storage medium and include computer programs used to cause a computer device to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.
[0304] The program product of the embodiments of this application may employ a portable compact disc read-only memory (CD-ROM) and include a computer program, and may run on an electronic device. However, the program product of this application is not limited thereto. In this document, the readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with a command execution system, apparatus, or device.
[0305] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a readable computer program. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with a command execution system, apparatus, or device.
[0306] Computer programs contained on readable media may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.
[0307] Computer programs for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java and C++, and conventional procedural programming languages such as C or similar languages. The computer program can execute entirely on the user's electronic device, partially on the user's electronic device, as a standalone software package, partially on the user's electronic device and partially on a remote electronic device, or entirely on a remote electronic device. In cases involving remote electronic devices, the remote electronic device can be connected to the user's electronic device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external electronic device (e.g., via the Internet using an Internet service provider).
[0308] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.
[0309] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.
[0310] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0311] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.
[0312] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.
[0313] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.
[0314] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.
Claims
1. A video encoding method, executed in a computer device, the method comprising: Obtain the control point motion vectors of at least two control points in the current block of a video frame; Each control point motion vector represents the motion vector of the corresponding control point between the video frame and the reference frame; The motion vector of the control point is optimized through at least one round of iteration; wherein each round of iteration includes: Based on the reference frame and the motion vector of the current control point, determine the prediction residual of the current block; Determine the gradient of the predicted residual with respect to each component of the motion vector of the current control point; The motion vector of the current control point is updated based on the gradient and the prediction residual.
2. The method of claim 1, wherein, Based on the gradient and the predicted residual, the motion vector of the current control point is updated, including: Based on the gradient and the prediction residual, the adjustment amount for the motion vector of the current control point is determined; Based on the adjustment amount, update the motion vector of the current control point.
3. The method of claim 2, wherein, The step of determining the adjustment amount for the motion vector of the current control point based on the gradient and the prediction residual includes: A system of linear equations is constructed using the gradient and the prediction residual, which indicates the mapping relationship from the gradient and the prediction residual to the adjustment amount of the motion vector of the current control point; Based on the binary bit width of the elements in the augmented matrix of the linear equation system, the linear equation system is eliminated to obtain the eliminated linear equation system. The augmented matrix of the eliminated linear equation system is a row echelon matrix. Based on the eliminated linear equations, the adjustment amount for the motion vector of the current control point is determined.
4. The method of claim 3, wherein, Based on the binary bit width of the elements in the augmented matrix of the linear equation system, the linear equation system is eliminated to obtain an eliminated linear equation system, including: Perform multiple rounds of elimination on the augmented matrix; each round of elimination includes: Select the first element from the current elimination column as the pivot; Calculate the first product of the first element and the second element, where the second element is any element in the augmented matrix except for the row containing the first element; Calculate the second product of the third element and the fourth element, wherein the third element is in the same column as the first element and in the same row as the second element, and the fourth element is in the same row as the first element and in the same column as the second element; Updating the second element based on the first product and the second product includes one of the following: Calculate the difference between the first product and the second product; perform a first right shift operation on the difference to obtain an updated second element, wherein the right shift bit is equal to the binary bit width of the first element; and Perform a first right shift operation on the first product to obtain a first result; perform a first right shift operation on the second product to obtain a second result; and use the difference between the first result and the second result as the updated second element.
5. The method of claim 4, wherein, Each round of elimination further includes: The sum of the binary bit width of the first element and the binary bit width of the second element is used as the first estimated bit width. The sum of the binary bit width of the third element and the binary bit width of the fourth element is used as the second estimated bit width; Determine whether one of the first estimated bit width and the second estimated bit width exceeds a preset bit width threshold; When it is determined that one of the first estimated bit width and the second estimated bit width exceeds the preset bit width threshold, a second right shift operation is performed on the one with the larger binary bit width between the first element and the second element; a second right shift operation is also performed on the one with the larger binary bit width between the third element and the fourth element; the right shift number of the second right shift operation is the difference between the binary bit width of the first element and the preset bit width threshold.
6. The method according to any one of claims 3 to 5, characterized in that, The coefficient part of the augmented matrix of the eliminated linear equation system is a diagonal matrix; The adjustment amount for the motion vector of the current control point is determined based on the eliminated linear equation system, including: Based on the diagonal matrix, the adjustment amount for each component of the motion vector of the current control point is calculated in parallel.
7. The method according to any one of claims 1 to 6, characterized in that, Determining the gradient of the predicted residual relative to each component of the motion vector of the current control point includes: The partial derivatives of the prediction residuals of the pixels in the current block with respect to each component of the motion vector of the current control point of the at least two control points are determined as the corresponding gradients.
8. The method according to any one of claims 1 to 7, characterized in that, The step of obtaining the control point motion vectors of at least two control points in the current block of a video frame includes: Construct a candidate list including at least one set of candidate motion vectors, where each set of candidate motion vectors is a control point motion vector of an encoded unit adjacent to the current block in the spatial or temporal domain; Calculate the rate-distortion cost for each group of candidate motion vectors in the candidate list; The candidate motion vectors with the lowest rate-distortion cost in the candidate list are determined as the control point motion vectors for the at least two control points.
9. A video encoding apparatus, comprising: The acquisition module is used to acquire the control point motion vectors of at least two control points in the current block of a video frame; Each control point motion vector represents the motion vector of the corresponding control point between the video frame and the reference frame; The processing module is used to perform at least one round of iterative optimization on the motion vector of the control point; wherein each round of iterative optimization includes: Based on the reference frame and the motion vector of the current control point, determine the prediction residual of the current block; Determine the gradient of the predicted residual with respect to each component of the motion vector of the current control point; The motion vector of the current control point is updated based on the gradient and the prediction residual.
10. A computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the method as described in any one of claims 1 to 8.
11. A computer device, comprising: include: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the method as described in any one of claims 1 to 8 according to the obtained program instructions.
12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the method as described in any one of claims 1 to 8.
