Video encoding method, video encoder, electronic device, and medium

Abstract

A video encoding method, a video encoder, an electronic device, and a medium, relating to the technical field of video processing. The video encoding method may comprise: acquiring a video stream to be encoded; for a current frame in the video stream, acquiring motion vectors and depth values respectively corresponding to a plurality of image blocks in the current frame; acquiring, on the basis of the depth values and the motion vectors, control parameters respectively corresponding to the plurality of image blocks, wherein each depth value represents a distance between content represented by the respective image block and a video acquisition apparatus, and each motion vector represents motion of the respective image block in the video stream; and encoding the current frame on the basis of the control parameters respectively corresponding to the plurality of image blocks, wherein the control parameters are used for representing visual saliency of the image blocks in the video stream, and are at least used for participating in discarding redundant information in the current frame during encoding.

Description

Video encoding methods, video encoders, electronic devices, and media Technical Field

[0001] This disclosure relates to the field of video processing technology, and in particular to a video encoding method, a video encoder, an electronic device, and a medium. Background Technology

[0002] With the rapid development of information technology, video transmission has become particularly important. Among these technologies, video encoding is a prerequisite for video transmission. It refers to the process of converting the original video file into another video file format through compression technology, thereby reducing the size of the original video data for easier storage and transmission.

[0003] Video encoding mainly involves data compression, such as removing redundant information from videos, and encoding videos so that they can be transmitted through various networks and devices.

[0004] Overview

[0005] A first aspect of this disclosure provides a video encoding method, wherein the method includes:

[0006] Obtain the video stream to be encoded;

[0007] For the current frame in the video stream, obtain the motion vector and depth value corresponding to each of the multiple image blocks in the current frame;

[0008] Based on the depth value and the motion vector, control parameters corresponding to each of the multiple image blocks are obtained; wherein, the depth value represents the distance between the content represented by the image block and the video acquisition device, and the motion vector represents the movement of the image block in the video stream;

[0009] The current frame is encoded based on the control parameters corresponding to each of the multiple image blocks;

[0010] The control parameters are used to characterize the visual attention of the image block in the video stream and at least to participate in the discarding of redundant information in the current frame during the encoding.

[0011] For example, encoding the current frame based on the control parameters corresponding to each of the plurality of image patches includes:

[0012] Based on the control parameters corresponding to each image block and the initial transformation parameters corresponding to each image block, each image block is transformed; wherein, the transformation is used to transform the image block from the spatial domain to the frequency domain;

[0013] Each of the converted image blocks is sequentially quantized and entropy encoded to obtain a precoded frame;

[0014] Based on the precoded frames and the control parameters, a video quality distortion model is constructed;

[0015] Based on the video quality distortion model, the encoding of the current frame is corrected.

[0016] For example, the control parameters include a first parameter and a second parameter, which are obtained by performing different calculations on the motion vector and the depth value;

[0017] The adjustment parameters and initial transformation parameters corresponding to each image block include:

[0018] Based on the first parameter corresponding to each image block, the initial transformation parameters corresponding to the image block are adjusted;

[0019] The step of constructing a video quality distortion model based on the precoded frame and the control parameters includes:

[0020] Based on the precoded frame, the first parameter, and the second parameter, a video quality distortion model for the current frame is constructed.

[0021] For example, obtaining the control parameters corresponding to each of the multiple image patches based on the depth value and the motion vector includes:

[0022] Based on the depth value, the first attention level of the image patch is obtained;

[0023] Based on the motion vector, the second attention level of the image patch is obtained;

[0024] The control parameters are obtained based on the first level of attention and the second level of attention;

[0025] Both the first attention level and the second attention level characterize the degree to which the image patch attracts visual attention.

[0026] For example, obtaining the second attention level of the image patch based on the motion vector includes:

[0027] Based on the maximum depth value of the current frame, obtain at least three depth value ranges corresponding to the current frame;

[0028] The second attention level is obtained based on the depth value range in which the depth value falls within the multiple depth value ranges;

[0029] Different depth value ranges correspond to different second attention levels.

[0030] For example, the method further includes: obtaining network parameters, the network parameters including the network bandwidth of the networks where the sending end and the receiving end are located, the bit error rate and packet loss rate of the communication channel between the sending end and the receiving end;

[0031] The step of constructing a video quality distortion model based on the precoded frame and the control parameters includes:

[0032] The video quality distortion model is constructed based on the precoded frame, the control parameters, and the network parameters.

[0033] For example, constructing the video quality distortion model based on the precoded frame, the modulation parameters, and the network parameters includes:

[0034] Based on the precoded frame and the current frame, obtain the distortion value corresponding to the current frame;

[0035] Based on the control parameters, the network parameters, and the distortion value, the video quality distortion model is constructed.

[0036] The distortion value is used to characterize the difference between the precoded frame and the current frame.

[0037] For example, obtaining the distortion value corresponding to the current frame based on the precoded frame and the current frame includes:

[0038] Obtain the depth map and texture map of the current frame; wherein the depth value is obtained based on the depth map, and the motion vector is obtained based on the texture map;

[0039] Obtain the first encoded map corresponding to the texture map and the second encoded map corresponding to the depth map in the precoded frame;

[0040] Based on the first encoded image and the texture image, a first distortion value is obtained;

[0041] Based on the second encoded map and the depth map, a second distortion value is obtained;

[0042] The distortion value is obtained based on the first distortion value and the second distortion value.

[0043] For example, correcting the encoding of the current frame based on the video quality distortion model includes:

[0044] Based on the video quality distortion model and the output bitrate of each image block in the precoded frame, the initial rate distortion model is corrected;

[0045] Based on the corrected rate-distortion model, the current frame is re-encoded to correct the encoding of the current frame.

[0046] For example, obtaining the control parameters corresponding to each of the multiple image patches based on the depth value and the motion vector includes:

[0047] Input the current frame into the target network;

[0048] Receive the control parameters corresponding to each image block in the current frame output by the target network;

[0049] The target network is obtained by self-supervised training of a preset network using multiple video samples as training samples. Each frame of the video sample carries multiple pixels with adjustment parameter labels. The adjustment parameter labels are obtained based on the motion vector and depth value of each predicted image block. The adjustment parameter labels are used to participate in the self-supervision during the training process. The predicted image block is obtained by image segmentation of each frame of the video sample.

[0050] For example, the target network includes:

[0051] The first feature extraction module is used to extract features from the texture map of the current frame to obtain texture features;

[0052] The second feature extraction module is used to extract features from the depth map of the current frame to obtain depth features;

[0053] The fusion module is used to fuse the texture features and the depth features to obtain fused features;

[0054] The prediction module is used to obtain the adjustment parameters of each pixel in the current frame based on the fused features;

[0055] The control parameters corresponding to the image block are obtained based on the control parameters of each pixel within the image block.

[0056] For example, the target network is obtained through the following steps:

[0057] Obtain the depth map and texture map of each frame of the video sample;

[0058] The depth map and the texture map are input into the preset network to obtain the prediction control parameters corresponding to multiple pixels of each frame of the video sample output by the preset network;

[0059] Based on the control parameter labels corresponding to multiple pixels and the predicted control parameters, the parameters of the preset network are updated multiple times to obtain the target network.

[0060] For example, the transformation of each image block based on the control parameters corresponding to each image block and the initial transformation parameters corresponding to each image block includes:

[0061] Obtain the frequency equalization matrix corresponding to each image block;

[0062] The initial transformation parameters are corrected based on the control parameters of the image patch and the frequency equalization matrix.

[0063] The transformation of each image block is completed based on the corrected initial transformation parameters.

[0064] For example, the modification of the initial transformation parameters based on the adjustment parameters of the image patch and the frequency equalization matrix includes:

[0065] Based on the frequency equalization matrix, the initial transformation parameters are corrected to obtain the first transformation coefficients;

[0066] Based on the preset matrix and the control parameters, the correction coefficient is obtained;

[0067] Based on the correction coefficient, the first transformation coefficient is corrected to obtain the corrected second transformation coefficient.

[0068] For example, obtaining the correction coefficient based on the preset matrix and the control parameters includes:

[0069] Based on the preset first coefficient and the maximum threshold corresponding to the control parameter, the control parameter is nonlinearly mapped;

[0070] Based on the preset matrix and the preset second coefficient, obtain the correction value for correcting the nonlinear mapping;

[0071] The correction coefficient is obtained based on the correction value and the result of the nonlinear mapping;

[0072] Wherein, both the preset first coefficient and the preset second coefficient are greater than 0 and less than 1.

[0073] For example, when obtaining the correction value for correcting the nonlinear mapping based on a preset matrix and preset second coefficients, the process includes:

[0074] Obtain the third coefficient corresponding to the nonlinear mapping;

[0075] Based on the maximum threshold, and using the preset first coefficient as the weight and the third coefficient as the exponent, the first parameter is nonlinearly mapped.

[0076] For example, obtaining the third coefficient corresponding to the nonlinear mapping includes:

[0077] The third coefficient is obtained based on the type of the current frame in the video stream.

[0078] For example, encoding the current frame based on the control parameters corresponding to each of the plurality of image patches includes:

[0079] Based on the control parameters corresponding to each of the multiple image blocks, each viewpoint image of the current frame is encoded to encode the video stream into a three-dimensional video stream;

[0080] The multi-viewpoint image of the current frame is an image composed of multiple viewpoint images with parallax from the same perspective.

[0081] A second aspect of this disclosure provides a video encoder, comprising a processing module configured to encode each frame of a video stream to be encoded, the video stream being acquired by a video acquisition device.

[0082] The processing module is specifically configured to perform the video encoding method described in any exemplary embodiment of the first aspect.

[0083] A third aspect of this disclosure provides an electronic device, wherein the electronic device includes a processor that, when running, executes the video encoding method described in any exemplary embodiment of the first aspect.

[0084] A fourth aspect of this disclosure provides a non-transitory computer-readable storage medium, wherein the medium stores, when executed by a computer's processor, a video encoding method as described in any exemplary embodiment of the first aspect.

[0085] The above description is merely an overview of the technical solution disclosed herein. In order to better understand the technical means of this disclosure and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this disclosure more apparent and understandable, specific embodiments of this disclosure are described below.

[0086] Brief description of the attached diagram

[0087] To more clearly illustrate the technical solutions in the embodiments or related technologies of this disclosure, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. It should be noted that the scale in the drawings is for illustration only and does not represent the actual scale.

[0088] Figure 1 shows a schematic diagram of the video transmission scenario in this embodiment;

[0089] Figure 2 shows a flowchart of the video encoding method;

[0090] Figure 3 illustrates a schematic diagram of image segmentation for each video frame in the video stream;

[0091] Figure 4 shows a schematic diagram of the process for determining the control parameters;

[0092] Figure 5 illustrates a specific example of the control parameters;

[0093] Figure 6 shows a schematic diagram of another process for obtaining control parameters;

[0094] Figure 7 shows a schematic diagram of the target network architecture;

[0095] Figure 8 illustrates the process of training the target network.

[0096] Figure 9 illustrates the process of adaptive adjustment of the control parameters;

[0097] Figure 10 illustrates the process of constructing a video quality distortion model;

[0098] Figure 11 shows a flowchart of the video encoding method for this specific vector.

[0099] Detailed description

[0100] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0101] In related technologies, video encoding is involved in many video transmission processes. For example, the video stream captured by the camera device needs to be encoded before it is transmitted to the receiving end, where the video stream is decoded and played back.

[0102] In some scenarios, video analysis at the receiving end may also be involved. This includes analyzing video footage to assess the actions, appearance, and correlation between the observed object and its surrounding environment. The observed object can be humans, animals, plants, or machinery. For example, the analysis could examine the degree of integration between an animal's actions, physical features, and its surroundings to record these characteristics for further research, such as conducting scientific studies or preparing datasets.

[0103] Therefore, in video analysis scenarios, it is necessary to ensure that the encoded video is conducive to subsequent video analysis. However, current video transmission not only faces a complex and ever-changing network transmission environment, but also fails to give sufficient attention and optimization to how to improve the contribution of encoded video to video analysis, such as ensuring the image clarity of key areas of interest in video analysis.

[0104] In view of this, this disclosure proposes a video coding method, a video encoder, an electronic device, and a storage medium. In this video coding method, when encoding each frame in a video stream, the motion vector and depth value of each image block in the current frame can be obtained first. Based on the motion vector and depth value, the control parameters corresponding to each of the multiple image blocks can be obtained. The encoding of each image block can then be controlled based on the control parameters, so that the encoded video has higher clarity in the key areas of interest for video analysis, thereby facilitating video analysis.

[0105] The control parameter can be used to characterize the visual attention of an image patch in a video stream. Visual attention can be understood as the degree of human eye perception of an image. Its visual attention is closely related to subsequent video analysis. This is because in video analysis, whether it is based on deep learning or manual analysis, it is based on human visual perception. Therefore, the visual attention in this embodiment can also be understood as the importance of the image patch to subsequent video analysis.

[0106] It should be noted that the video stream targeted in this embodiment can be acquired by a video acquisition device, which can be a binocular image acquisition device. In this way, the video stream can contain binocular depth and binocular texture information, and each frame contains depth information. Therefore, the depth value required to calculate the control parameters can be obtained based on the depth information, which can improve the accuracy of the control parameters, thereby improving the accuracy of visual perception of each image block, and thus enhancing the contribution to subsequent video analysis.

[0107] The video encoding method, video encoder, electronic device, and storage medium in the embodiments of this disclosure will now be described by way of example with reference to the accompanying drawings.

[0108] In one embodiment, a video encoding method is provided. Please refer to Figures 1 and 2. Figure 1 shows a schematic diagram of the video transmission scenario of this embodiment, and Figure 2 shows a flowchart of the video encoding method. As shown in Figures 1 and 2, the video encoder in this embodiment can be applied to the video sending end. The video sending end includes a video encoder, which can be an encoder of the H.265 / HEVC standard.

[0109] As shown in Figure 1, a video transmission scenario may include a binocular image acquisition device 11. This device may include two cameras, which can capture video of the scene from different perspectives. Each frame in the acquired video stream may include texture information and depth information. The depth information reflects the distance of each scene element in the video frame from the binocular image acquisition device, and the texture information reflects the texture of each pixel in the video, such as the RGB values ​​of each pixel. Here, RGB represents the three color channels of a pixel: R represents red, G represents green, and B represents blue.

[0110] In this embodiment, the texture map can be understood as the color map of the current frame, that is, an image containing RGB values.

[0111] Of course, in an exemplary embodiment, the acquired video stream may also be acquired by a non-binocular image acquisition device, such as a two-dimensional video. In this case, the video stream can be processed using deep learning technology to extract depth information and texture information from the original video stream.

[0112] The video stream acquired by the binocular image acquisition device can be sent to the video encoder 22, which encodes each frame of the video stream and outputs the encoded video stream to the receiving end 33. The video encoder 22 can be built into the binocular image acquisition device 11 or be independent of the binocular image acquisition device 11.

[0113] When the encoded video stream is output to the receiving end, it can be transmitted through the communication channel 44 between the receiving end and the video encoder. This communication channel can be a Bluetooth channel, a broadband network channel, a satellite channel, etc., where the broadband network can include a local area network, a wide area network, and a metropolitan area network.

[0114] Among them, the quality of the communication channel, such as bandwidth, will affect the efficiency of video transmission, such as the packet loss rate and time of video transmission, thereby affecting the clarity and transmission rate of the video received by the receiving end.

[0115] In this embodiment, the encoding of the video stream can also be modified based on the calculated control parameters of each image block, so that the encoded video stream is adapted to the quality of the communication channel, balancing the video transmission speed and the video packet loss rate, thereby ensuring the clarity of the video to facilitate subsequent video analysis.

[0116] As shown in Figure 2, taking the video stream acquired by the binocular image acquisition device as an example, the video encoding method in this embodiment mainly includes the following steps:

[0117] Step S100: Obtain the video stream to be encoded;

[0118] Step S200: For the current frame in the video stream, obtain the motion vectors and depth values ​​corresponding to each of the multiple image patches in the current frame;

[0119] Step S300: Based on depth values ​​and motion vectors, obtain the control parameters corresponding to each of the multiple image patches;

[0120] Step S400: Encode the current frame based on the control parameters corresponding to each of the multiple image blocks;

[0121] The depth value represents the distance between the content represented by the image block and the binocular image acquisition device, and the motion vector represents the motion of the image block in the video stream.

[0122] Among them, the control parameters are used to characterize the visual attention of image patches in the video stream, and at least to participate in the discarding of redundant information in the current frame during encoding.

[0123] In this embodiment, since the video stream is acquired by a binocular image acquisition device, each frame of the video stream contains texture information and depth information. Therefore, a depth map and a texture map can be obtained for each frame of the video stream. For ease of explanation, a frame in the video stream will be referred to as a video frame below.

[0124] In acquiring the depth map, methods such as stereo vision and optical flow can be used to extract the depth map from the video frame. Each pixel in the depth map represents the distance from that point to the binocular image acquisition device.

[0125] In obtaining the texture map, the video frames can be filtered and denoised first, and then the texture of the video frames can be extracted to obtain the texture map. Each pixel of the texture map includes RGB values. The texture map can extract the contour features and detail features of each scene content in the video frame.

[0126] In this embodiment, when encoding the video stream, each frame of the video stream can be encoded. When encoding each frame, the depth map and texture map of the frame can be obtained. Based on the depth map, the depth value of each pixel in the frame can be obtained, and based on the texture map, the RGB value of each pixel can be obtained, thereby extracting the outline and details of the frame.

[0127] When encoding the current frame of a video stream, the frame can be segmented into multiple image blocks. These image blocks can have the same size, for example, any one of 4×4, 8×8, 16×16, or 32×32. This size represents the length and width of the image block. For example, 4×4 means the image block has a length of 4 pixels P and a width of 4 pixels P. Generally, the size of the image block can be 2... n ×2 n This means that the image patch has the same length and width, and contains 2 n P pixels.

[0128] For example, as shown in FIG3, FIG3 illustrates a schematic diagram of image segmentation for each video frame in a video stream. As shown in FIG3 (1), the video frame can be uniformly divided into multiple image blocks PP with a size of 4×4. Each image block PP has a size of 4×4.

[0129] In one exemplary embodiment, the sizes of different image blocks in a video frame can be different. As shown in Figure 3(2), the video frame can be divided into multiple image blocks of different sizes, such as 4×4, 8×8, and 16×16. In this embodiment, areas with roughly the same content in the video frame can be divided into a larger image block. For example, as shown in Figure 3(2), the video frame contains a large area with the same content, so this area A1 can be divided into a larger size of 8×8. Of course, Figure 3 is only an example. In some embodiments, for large image areas with the same content, they can also be divided into 16×16 blocks.

[0130] In step S200, for the current frame to be encoded, the current frame can be divided into multiple image blocks. Then, the texture map and depth map of the current frame can be obtained. Based on the texture features in the texture map within the multiple image blocks, inter-frame prediction can be performed to obtain the motion vector of each image block. The motion vector can refer to the motion state of the image block in multiple frames. The motion vector includes the motion direction and motion amplitude of the image block in multiple frames. Multiple frames can refer to multiple consecutive frames whose playback time is after the current frame, and multiple consecutive frames whose playback time is before the current frame.

[0131] Motion vectors can characterize the intensity of motion of the content represented by an image patch. The intensity of motion is related to the degree to which it attracts human visual attention; for example, content that moves further away will attract more visual attention. Therefore, the visual attention of each image patch in the current frame is related to the motion vector of the image patch.

[0132] Specifically, the depth value of an image patch can be determined based on the depth values ​​of multiple pixels contained within it in the depth map. This depth value can be the average of the depth values ​​of the pixels within the image patch. The depth value of an image patch reflects the distance of the content it represents from the binocular image acquisition device; it can also be understood as the depth of field of the image patch. This depth value is also related to its visual attention-grabbing ability; for example, content closer to the binocular image acquisition device will attract more visual attention. Therefore, the visual attention of each image patch in the current frame can be related to its depth value.

[0133] In step S300, after obtaining the depth value and motion vector of the image patch, since the visual attention of the image patch is related to the depth value and motion vector of the image patch, the control parameters corresponding to each image patch can be obtained by combining the depth value and motion vector. The control parameters can characterize the importance of the image patch to visual attention, or can be understood as the control parameters can characterize the degree of attraction of the image patch to human vision, thereby characterizing the importance of the image patch to video analysis.

[0134] For example, the higher the visual attention represented by the control parameter, the higher its attractiveness to human vision, the more important the image patch is to subsequent video analysis. The content contained in the image patch will be the focus of analysis in subsequent video analysis, that is, the content contained in the image patch can be the key content in video analysis.

[0135] In an exemplary embodiment, the control parameter can be represented by a numerical value, and its range can be 0 to 1, that is, the control parameter can be normalized, and the normalization can be a value between 0 and 1; or, the range of the control parameter can be between 0 and 10, specifically, the control parameter can be a positive integer between 0 and 10.

[0136] The larger the value of the control parameter, the higher the visual attention it receives. In other words, the larger the value of the control parameter, the more important the image patch is to video analysis, and the more visual attention it will attract.

[0137] In an exemplary embodiment, after the range of control parameters is set, multiple fixed values ​​within the range of control parameters can be set. When determining the control parameters of an image block, a target fixed value corresponding to the depth value and motion vector can be selected from the multiple fixed values ​​as the control parameters of the image block based on the depth value and motion vector of the image block.

[0138] For example, assuming the range of the adjustment parameter is 0 to 1, and the selected fixed values ​​are 0.2, 0.4, 0.6 and 0.8 respectively, then the corresponding value among 0.2, 0.4, 0.6 and 0.8 can be selected as the adjustment parameter according to the depth value and motion vector of the image block.

[0139] Of course, in an exemplary embodiment, the control parameters can also be dynamically determined within the range of control parameters based on depth values ​​and motion vectors. Different combinations of depth values ​​and motion vectors can correspond to control parameters of different sizes, so that the control parameters are dynamically distributed within the range of control parameters. In this way, the correlation between the control parameters and the depth values ​​and motion vectors of the image blocks can be improved, and the precision of the control parameters can be improved.

[0140] The range of control parameters can be determined based on the content targeted by the video analysis. For example, if the video analysis is to analyze dynamic features, such as the feeding characteristics of animals, it needs to analyze the amount of food consumed, feeding preferences, feeding actions, etc. The analysis has many dimensions. Therefore, the range of control parameters can be set larger and the number of fixed values ​​can be set more, so that the encoding of the current frame can be adapted to the video analysis and the encoding precision can be improved.

[0141] In step S400, since the control parameters of the image blocks are related to visual attention, each image block can be encoded according to the control parameters. In actual encoding, the image blocks can be sequentially subjected to frequency conversion, quantization, and entropy encoding. The control parameters can adjust the frequency conversion of the image blocks. For example, the control parameters can adjust the high-frequency coefficients of the image blocks, thereby preserving some of the high-frequency features of the image blocks.

[0142] In this embodiment, frequency conversion refers to converting an image patch from the spatial domain to the frequency domain. This conversion can maintain the size of the image patch unchanged. After frequency conversion, the upper left corner of the image patch represents low-frequency features, and the lower right corner represents high-frequency features. Since most of the energy in the current frame is concentrated in the low-frequency portion, after converting the image patch into frequency coefficients, high-frequency coefficients are generally discarded while low-frequency coefficients are retained to reduce the amount of data required for the image patch without sacrificing too much image quality.

[0143] In this embodiment, in order to ensure the clarity of the image blocks that are the focus of subsequent video analysis, the high-frequency coefficients can be adjusted according to the control parameters of the image blocks. For example, some high-frequency coefficients can be retained to preserve the details of the image blocks and improve image quality.

[0144] For example, a larger control parameter for an image patch indicates higher visual attention, which can enhance high-frequency coefficients and thus preserve the details of the image patch; a smaller control parameter for an image patch indicates lower visual attention, which can suppress high-frequency coefficients and thus discard the high-frequency part of the image patch, thereby reducing the amount of image patch data and achieving maximum compression.

[0145] Among them, the high-frequency coefficients can be the coefficients originally stored in the encoder. For example, if the encoder is H.265 / HEVC, then the high-frequency coefficients can be the coefficients under the H.265 / HEVC standard.

[0146] Among them, enhancing the high-frequency coefficient based on the control parameters can refer to increasing the high-frequency coefficient under the H.265 / HEVC standard, while suppressing the high-frequency coefficient based on the control parameters can refer to decreasing the high-frequency coefficient under the H.265 / HEVC standard.

[0147] The high-frequency components of the image block after frequency conversion can be referred to as the redundant information of the image block. Thus, discarding the high-frequency components means discarding the redundant information. Therefore, the process of adjusting the high-frequency coefficients based on the control parameters can be understood as adjusting the discard rate of the redundant information of the image block based on the control parameters. That is, the control parameters participate in the discarding of the redundant information of the image block. More specifically, in conjunction with the above embodiments, the control parameters guiding the discarding of redundant information can specifically occur in the frequency conversion stage of the image block, and are used to determine the correction amount of the high-frequency coefficients, that is, to suppress the high-frequency coefficients or enhance the high-frequency coefficients.

[0148] Below, we will provide an example to illustrate this.

[0149] Assuming video analysis focuses on analyzing animal feeding behavior, a binocular image acquisition device can capture the animal's feeding process. During filming, the device tracks the animal's feeding motion, ensuring that each frame of the video stream contains images of the animal's mouth and surrounding area. When encoding this video stream, the current frame can be divided into multiple image blocks. Since video analysis prioritizes the analysis of the animal's mouth and surrounding area, image blocks containing these areas will receive higher visual attention. When determining the control parameters for these image blocks, the mouth and surrounding area exhibit larger motion vectors, characterized by frequent changes in direction and significant amplitude of movement. Furthermore, the depth values ​​of these areas indicate their proximity to the binocular image acquisition device. Therefore, the control parameters for image blocks in these areas can characterize their ability to attract significant visual attention. In this way, the image blocks in and around the mouth have larger control parameters. When performing frequency conversion on these image blocks, the high-frequency coefficients of these image blocks can be preserved and enhanced by controlling the parameters, thereby retaining more detailed features. When performing frequency conversion on the remaining parts (image blocks other than those in and around the mouth), the high-frequency coefficients of these image blocks can be suppressed by controlling the parameters, thereby discarding more redundant information and reducing the total amount of data required.

[0150] In this way, after encoding the current frame, the receiving end decodes the current frame. The decoded frame contains a clearer image of the animal's mouth and the surrounding area with more details, which is helpful for feeding analysis and improves the accuracy of feeding analysis.

[0151] In this embodiment, control parameters are determined by combining the depth values ​​and motion vectors of each image block in the video frame. The depth value reflects the distance of the content represented by the image block from the viewer and is closely related to the degree of visual attraction of the image block. The motion vector reflects the intensity of the movement of the image block and is also closely related to the degree of visual attraction of the image block. Therefore, the control parameters determined based on these two factors can more realistically and accurately reflect the visual perception of the image block. During encoding, more image details can be retained for image blocks with high visual attention (larger control parameters), while more redundant information can be discarded for image blocks with low visual attention (smaller control parameters). This improves the clarity of image blocks with high visual attention, which is beneficial for subsequent video analysis and enhances the contribution of video encoding to video analysis. In addition, it can also ensure the compression level of video encoding to reduce the network bandwidth consumption during video transmission, thereby ensuring video transmission efficiency.

[0152] In an exemplary embodiment, the video encoding method described in steps S100-S400 above can be used to generate a three-dimensional video stream. Thus, when encoding the current frame, each viewpoint image of the current frame can be encoded based on the control parameters corresponding to each of the multiple image blocks, so as to encode the video stream into a three-dimensional video stream.

[0153] The multi-viewpoint image of the current frame is an image composed of multiple viewpoint images with parallax from the same perspective.

[0154] Among them, a three-dimensional video stream refers to a video stream that contains stereoscopic spatial information. For example, each frame of the video stream may include a multi-view image, which is an image that is merged from multiple viewpoint images with parallax from the same perspective. In this way, the image of each viewpoint in each frame can be encoded.

[0155] Specifically, the video image of each viewpoint contains depth information and texture information. During encoding, each viewpoint image of each frame needs to be encoded. The video image of each viewpoint to be encoded can be called the current viewpoint image in the current frame. When the current viewpoint image is in the current frame, each viewpoint image can be divided into multiple image blocks according to the process described in steps S100-S400 above. Then, according to the control parameters corresponding to the multiple image blocks of each viewpoint image, the depth map and texture map of the current viewpoint image are encoded. The results of the two encodings (the texture map corresponds to the first encoded map, and the depth map corresponds to the second encoded map) are fused to form the encoded stream of the current viewpoint image.

[0156] This involves merging the encoded streams of multiple viewpoint images in the current frame to form the encoded stream of the current frame, thereby generating a 3D video stream.

[0157] The control parameters can be applied to the encoding of the depth map corresponding to each viewpoint image, and also to the encoding of the texture map corresponding to each viewpoint image.

[0158] The texture map can be understood as the color map of the current frame, that is, an image containing RGB values.

[0159] In some embodiments, please refer to Figure 4, which illustrates the process of determining control parameters. As shown in Figure 4, when determining control parameters, a first level of attention can be determined based on the depth value, and a second level of attention can be determined based on the motion vector. Then, control parameters are obtained based on the first level of attention and the second level of attention.

[0160] Both the first attention level and the second attention level characterize the degree to which an image patch attracts visual attention.

[0161] As shown in Figure 4, the motion vector of each image patch can be obtained based on the texture map of the current frame, and the depth value of each image patch can be obtained based on the depth map of the current frame. The motion vector can characterize the intensity of motion of the content represented by the image patch. Therefore, a first attention level can be obtained based on the motion vector. This first attention level can be used to describe the degree to which the intensity of motion of the image patch attracts visual attention.

[0162] The depth value can characterize the distance between the content represented by the image patch and the observer. Therefore, a second attention level can be obtained based on the depth value. The second attention level can be used to characterize the degree to which the distance between the image patch and the observer attracts visual attention.

[0163] The first and second attention levels can be represented by numerical values, with different values ​​indicating different levels, that is, different degrees of visual attention.

[0164] As shown in Figure 5, Figure 5 illustrates a specific example of the control parameters. As shown in Figure 5, the first attention level and the second attention level can be represented as 0 and positive integers. For example, the first attention level can include three attention levels: 0, 1, and 2. The second attention level can include three attention levels: 0, 1, and 2. When the attention level is 0, the degree of attraction to visual attention is the lowest, and when the attention level is 2, the degree of attraction to visual attention is the highest.

[0165] In one exemplary embodiment, when determining the first level of attention, the first level of attention can be obtained based on the vector range in which the motion vector is located. For example, as shown in Figure 5... V represents the motion vector. k Indicating the first level of attention, as mentioned above, the motion vector can include the direction and magnitude of motion, which can be based on... The scope of the location determines the first level of attention; different scopes correspond to different levels of first level of attention.

[0166] in, A higher value indicates more intense exercise, and its primary attention level is V. k The value can be higher.

[0167] For example, as shown in Figure 5, assume that the image patch When the length is greater than or equal to ε·l (ε is the set threshold, and l is the length of the diagonal), it indicates the presence of a large and rapid movement, and the first level of attention, V, can be determined. k It is 2; if When the length is less than ε·l, the first level of attention V can be determined. k =1; otherwise, the image patch A length of 0 indicates an area with a relatively flat texture or a static background, thus defining the first attention level as V. k It is 0.

[0168] In one exemplary embodiment, when determining the second attention level, the second attention level can be obtained based on the depth range in which the depth value is located. Specifically, at least three depth value ranges corresponding to the current frame can be obtained based on the maximum depth value of the current frame; the second attention level can be obtained based on the depth value range in which the depth value is located among the multiple depth value ranges.

[0169] The depth value can characterize the distance between the content represented by the image patch and the observer (the observer is a binocular image acquisition device, which can be approximated as a video viewer). The larger the depth value, the farther away it is from the observer. The second level of attention can be determined based on the distance.

[0170] As shown in Figure 5, the maximum depth value of the current frame can refer to the depth value of the pixel farthest from the observer in the current frame. Through this maximum depth value, the depth values ​​of each pixel in the current frame can be divided into multiple depth value intervals, resulting in multiple depth value ranges.

[0171] For example, as shown in Figure 5, assume the maximum depth value is D. max To divide the depth into three intervals, we can set two constants α1 and α2, where 0 < α1 < α2 < 1. Thus, the three depth ranges are [0, α1·D]. max ]、(α1·D max ,α2·D max ) and (α2·D max ,1), the three depth value ranges correspond to the three second attention levels 0, 1 and 2 respectively.

[0172] After obtaining the depth value corresponding to each image block, the target depth value range where the depth value of the current image block is located can be determined within multiple depth value ranges, and the second attention level corresponding to the target depth value range can be determined as the second attention level corresponding to the image block.

[0173] For example, as shown in Figure 5, assuming the second attention level is D, the depth value of the image patch is... exist Located in [0, α1·D max When D is 2, it means the distance to the observer is closest; when D is located at (α1·D) max ,α2·D max When D is at (α2·D), D is 1, indicating that the distance to the observer is relatively close; when D is at (α2·D), D is 1. max When ,1), D is 0, indicating that the distance from the observer is the farthest.

[0174] The symbol “·” represents the multiplication operation.

[0175] The number and range of values ​​for the first and second attention levels can be the same, so that the depth value and motion vector can be mapped to the same space, which will facilitate the determination of subsequent control parameters.

[0176] Of course, the values ​​of 0, 1, and 2 for the first and second attention levels mentioned above are just examples. In practice, more values ​​can be used to make the visual attention of image blocks more refined.

[0177] In an exemplary embodiment, the control parameters may include a first parameter and a second parameter, which may be obtained by performing different calculations on a first attention level and a second attention level.

[0178] For example, assuming the first parameter is L and the second parameter is F, the first parameter L and the second parameter F can be obtained according to the following formulas (1) and (2): L = V && D (Formula (1)) F = V + D (Formula (2))

[0179] Where V represents the first level of attention and D represents the second level of attention.

[0180] Thus, the first parameter can be represented as the priority of visual attention to an image patch, which can be understood as the sensitivity of vision to an image patch, for example, whether an image patch can quickly attract visual attention. The second parameter can be represented as the importance of an image patch to visual attention, which can be the degree to which vision maintains attention to an image patch, for example, whether an image patch can attract visual attention for a long time.

[0181] Therefore, the visual attention to an image can be described from different aspects of visual perception using the first and second parameters.

[0182] In this way, different stages of video encoding can be adjusted using the first and second parameters. For example, the second parameter can be primarily applied to the frequency transformation of image blocks, while the combination of the first and second parameters can be used to adjust the compression level of image blocks, thus achieving more refined encoding adjustments. As shown in Figure 5, when the first parameter L is 0 and the second parameter F is 0, it indicates that the visual attention of the image block is 0. In subsequent encoding, all redundant information of the image block can be discarded, thereby improving the compression ratio. When the first parameter L is 1 and the second parameter F is 3, it indicates that the visual attention of the image block is relatively high. In subsequent encoding, some redundant information can be retained to improve image clarity.

[0183] For example, if the values ​​of the first attention level and the second attention level are both 0, 1 and 2, the value of the second parameter F can be 0, 1, 2, 3 and 4.

[0184] In some other embodiments, the robustness and generalization of the determination of the control parameters can be improved, so that the same encoder can quickly encode multiple video streams and ensure robustness to encoding video streams containing different content. The control parameters of each image block can be obtained by using deep learning.

[0185] Referring to Figures 6 and 7, Figure 6 shows another process for obtaining control parameters, and Figure 7 shows a schematic diagram of the target network architecture. As shown in Figures 6 and 7, multiple video samples can be used as training samples to train the preset network, thereby obtaining the target network. Then, the current frame can be input into the target network to obtain the control parameters corresponding to each image block in the current frame output by the target network.

[0186] In this process, multiple pixels in each frame of the video sample carry control parameter labels. The control parameter labels are obtained based on the motion vector and depth value of each predicted image patch. The control parameter labels are used to participate in self-supervision during the training process. The predicted image patch is obtained by image segmentation of each frame of the video sample.

[0187] In this embodiment, multiple video samples can be obtained in advance. The different video samples can be video samples taken for different scenes and different content.

[0188] In one exemplary embodiment, the video samples may be used for the same or different video analyses. For example, they may include video samples analyzing the process of animals eating, video samples analyzing the physical characteristics of animals and plants, and video samples analyzing animal movement.

[0189] The video samples were also acquired using a binocular image acquisition device, and they contain depth and texture information.

[0190] Each frame of the video sample can be divided into multiple predicted image blocks, and the division method can refer to the way the current frame is divided into multiple image blocks.

[0191] The predicted image patch and the current frame image patch can have the same or different sizes. For example, it is not required that the predicted image patch and the current frame image patch have the same size, thus reducing the difficulty of preparing the dataset.

[0192] In this process, each predicted image block in each frame of the video sample carries a control parameter label. The process of obtaining the control parameter label can refer to the process of obtaining the control parameter described above. For example, it can be obtained based on the motion vector and depth value of the predicted image block.

[0193] Among them, the control parameter labels can be manually annotated and can be used in the self-supervision process. This self-supervision refers to: during the training process, a loss function is constructed using the control parameter labels and the control parameters predicted by the preset network, and the parameters of the preset network are updated based on the loss function.

[0194] In one exemplary embodiment, when obtaining the motion vectors of predicted image blocks, video samples can be encoded. Since the encoding process involves inter-frame coding and intra-frame coding, and inter-frame coding predicts the motion of image blocks, motion information for each predicted image block is saved during encoding. This motion information can include motion vectors. Therefore, the motion vectors of each predicted image block can be directly obtained from the intermediate encoded file. This allows the use of existing encoders to obtain motion vectors without separate prediction, thus improving the efficiency of annotating the motion vectors of predicted image blocks.

[0195] In an exemplary embodiment, the process of determining the control parameters involved in FIG4 can be converted into a computer-readable program and burned into the processor of an electronic device. After obtaining the motion vector and depth value of each predicted image block, the motion vector and depth value of each predicted image block can be input into the processor, thereby automatically giving the control parameter label of each predicted image block. Thus, the automatic labeling of the control parameters of each predicted image block can be completed, thereby avoiding manual labeling and improving the collection efficiency of training samples.

[0196] In each training process, the training of the preset network can be supervised by adjusting the parameter labels, so as to update the parameters in the preset network. After multiple training and updates, the target network can be obtained.

[0197] The training can end when the network reaches convergence.

[0198] In an exemplary embodiment, as shown in FIG8, a schematic diagram of the process of training the target network is illustrated. As shown in FIG8, the depth map and texture map of each frame of the video sample can be obtained first; then, the depth map and texture map are input into the preset network to obtain the prediction control parameters corresponding to multiple pixels of each frame of the video sample output by the preset network; then, based on the control parameter labels and prediction control parameters corresponding to multiple pixels, the parameters of the preset network are updated multiple times to obtain the target network.

[0199] The process of obtaining the depth map and texture map of each frame in the video sample can be referred to the process of obtaining the depth map and texture map of the current frame as described above, and will not be repeated here.

[0200] In this process, depth maps and texture maps can be input into a preset network. The preset network can obtain the depth value of the predicted image patch based on the depth map and the motion vector of the predicted image patch based on the texture map. Then, based on the depth value and motion vector, the prediction control parameters of the predicted image patch can be predicted.

[0201] Next, the prediction control parameters and control parameter labels of the image patches can be used to construct a loss function.

[0202] In one example, the loss function can be represented by the following formula (3):

[0203] In formula (1), sig(.) represents the sigmod activation function, and sig(F) can be regarded as a representation of the control parameter; Let F represent the predicted control parameter, and let f(x) represent the control parameter label. f(x) represents a common loss function, including L1 loss function corresponding to the first parameter, L2 loss function corresponding to the second parameter, and mixed cross-entropy loss function.

[0204] In formula (1), the control parameters may include a first parameter and a second parameter, which may be obtained by performing different calculations on the first attention level and the second attention level.

[0205] In formula (1), β represents the weight of the first parameter, and (1-β) represents the weight of the second parameter.

[0206] The value of β can be set based on experience; β can be a value between 0 and 1.

[0207] In this embodiment, the preset network needs to determine the prediction control parameters for each pixel in the video sample, which may not involve the division of image blocks. Therefore, when constructing the loss function, it can be done at the pixel level, that is, the loss function is constructed using the prediction control parameters and control parameter labels for each pixel. Specifically, the output of the preset network can be the prediction control parameters corresponding to each pixel.

[0208] In this way, the control parameter label corresponding to each pixel in the predicted image patch can be the control parameter label corresponding to the predicted image patch.

[0209] In this embodiment, training can be terminated when the value of the loss function LOSS is less than or equal to a preset threshold, thereby using the preset network at the end of training as the target network.

[0210] The structure of the target network in this embodiment can be consistent with the structure of the preset network. As shown in Figure 7, the target network in this embodiment may include a first feature extraction module 55, a second feature extraction module 66, a fusion module, and a prediction module.

[0211] As shown in Figure 7, the first feature extraction module 55 and the second feature extraction module 66 can be connected to the fusion module respectively, and the output of the fusion module can be connected to the prediction module.

[0212] The first feature extraction module is used to extract features from the texture map of the current frame to obtain texture features;

[0213] The second feature extraction module is used to extract features from the depth map of the current frame to obtain depth features;

[0214] The fusion module is used to fuse texture features and depth features to obtain fused features;

[0215] The prediction module is used to obtain the control parameters of each pixel in the current frame based on the fused features.

[0216] The control parameters corresponding to the image block are obtained based on the control parameters of each pixel within the image block.

[0217] As shown in Figure 7, the texture map of each frame in the video sample can be input into the first feature extraction module. The first feature extraction module can extract features from the texture map to obtain texture features. Specifically, the first feature extraction module may include multiple texture units, such as a global texture map unit, texture feature units of different sizes, a first combination unit, and a second combination unit.

[0218] Among them, the global texture map unit can be a simple convolutional neural network to extract global features from the texture map; texture feature units of different sizes can have different convolutional kernel sizes to extract texture features under different receptive fields. The texture feature unit can include hierarchical pooling, convolution, hierarchical upsampling, etc., to obtain feature maps of different levels.

[0219] The first combination unit can be used to fuse feature maps at different levels. This fusion can stitch feature maps at different levels together. The second combination unit can perform convolution on the feature map fused by the first combination unit. For example, the second combination unit can be a fully connected layer, which can perform a fully connected operation on the fused feature map to obtain texture features with stronger expressive power.

[0220] As shown in Figure 7, the depth map of each frame in the video sample can be input into the second feature extraction module. The second feature extraction module can extract features from the depth map to obtain depth features. Specifically, the first feature extraction module may include multiple depth units, such as a depth compression unit and a depth extraction unit.

[0221] As shown in Figure 7, the depth compression unit can be used to perform simple multiple convolution operations on the depth map. The multiple convolution operations can be performed iteratively, and the kernel sizes for different convolution operations can be different. In this way, the feature map output by the depth compression unit can be called the depth compressed feature.

[0222] The depth extraction unit can perform pooling operations on the features output by the depth compression unit. This pooling operation can be a global pooling operation, thereby obtaining better depth features.

[0223] As shown in Figure 7, the fusion module can fuse depth features and texture features. This fusion can be a dot product operation, such as multiplying depth features and texture features by matrices; or it can be an addition operation, such as adding depth features and texture features by matrices; or it can be a concatenation operation.

[0224] As shown in Figure 7, the prediction module performs fully connected processing on the fusion features output by the fusion module to obtain the prediction control parameters corresponding to each pixel in each frame of the video sample.

[0225] In this way, after using the target network to predict the modulation parameters of each pixel in the current frame to be encoded, since the encoding is performed on the image blocks of the current frame, the modulation parameters corresponding to each image block in the current frame can be the average value of the modulation parameters of each pixel in that image block.

[0226] In this embodiment, the target network is used to obtain the control parameters of each image patch, which can improve robustness. This part of the network can achieve self-supervised training without the need for manual labeling of training samples, thereby reducing the difficulty of dataset collection.

[0227] In this embodiment, after obtaining the control parameters corresponding to each image block, the encoding of the current frame can be corrected based on the control parameters. Specifically, the frequency transformation process of the image blocks during the encoding process can be corrected, and in some examples, the compression rate of the encoding can be corrected, thereby balancing the video transmission efficiency and the clarity required for video analysis.

[0228] Referring to Figure 8, which illustrates the process of encoding the current frame based on control parameters, the specific steps include:

[0229] Step S401: Based on the control parameters and initial transformation parameters corresponding to each image block, transform each image block;

[0230] Step S402: Quantize and entropy encode each converted image block sequentially to obtain a precoded frame;

[0231] Step S403: Construct a video quality distortion model based on the precoded frames and control parameters;

[0232] Step S404: Correct the encoding of the current frame based on the video quality distortion model.

[0233] The transformation is used to convert the image patch from the spatial domain to the frequency domain, that is, the transformation is the frequency transformation described in the above embodiments.

[0234] In step S401, the initial transformation parameters corresponding to the image block can be adjusted based on the control parameters corresponding to the image block, and the image block can be frequency transformed based on the adjusted initial transformation parameters.

[0235] The frequency transformation process can be a discrete cosine transform, and the initial transformation parameters can be the transformation coefficients in the encoder according to its encoding standard. The initial transformation parameters for each image block can be independent, which means that image blocks of different sizes correspond to different initial transformation parameters.

[0236] In this process, the control parameter can be used as a correction value for the initial transformation parameter to correct the initial transformation parameter, thereby obtaining the adjusted initial transformation parameter. Then, the image block can be frequency converted according to the adjusted initial transformation parameter.

[0237] As illustrated in the above embodiments, a larger control parameter indicates higher visual attention, in which case high-frequency components can be enhanced; conversely, a smaller control parameter indicates lower visual attention, in which case high-frequency components can be suppressed. This allows limited coding resources to be preferentially allocated to visually high-attention areas in the video stream, ensuring subjective image quality and achieving priority coding of key attention areas.

[0238] In an exemplary embodiment, the transformation parameters can be adjusted only for the high-frequency coefficients. Specifically, to avoid severe block artifacts caused by manually selecting transformation parameters, reduce the burden on the sampling adaptive compensation filter, and lower the coding computation complexity, this embodiment uses a dynamic frequency equalization matrix with adaptive characteristics to adjust the initial transformation coefficients, thereby optimizing the existing coding standard, rather than using the existing coding standard alone.

[0239] Referring to Figure 9, which illustrates the adaptive adjustment process of the control parameters, the initial transformation parameters can be corrected based on the control parameters and the frequency equalization matrix. Specifically, the frequency equalization matrix W corresponding to each image block can be obtained. F Next, based on the image patch's modulation parameter F and frequency equalization matrix W... F The initial transformation parameter C is corrected; then, the transformation of each image block is completed based on the corrected initial transformation parameter C'.

[0240] In this embodiment, each image block corresponds to an independent frequency equalization matrix, and image blocks of different sizes correspond to different frequency equalization matrices. The dimension of the frequency equalization matrix can be related to the size of the image block. For example, the frequency equalization matrix can be represented as the following matrix:

[0241] Next, the initial transformation parameters C can be corrected based on the control parameter F and the frequency equalization matrix. Specifically, the control parameter F, the frequency equalization matrix, and the initial transformation parameters can be subjected to corresponding operations, such as multiplication or convolution, to obtain the transformed parameters.

[0242] As shown in Figure 9, the frequency equalization matrix and the control parameters can be used together to control the initial transformation parameter C, thereby obtaining the corrected transformation parameter C'.

[0243] The frequency equalization matrix has the same dimension as the image patch size, and each parameter in the frequency equalization matrix corresponds to a pixel in the image patch.

[0244] By adjusting the parameters, the values ​​of each parameter in the frequency equalization matrix can be changed, thereby enhancing or suppressing high-frequency components in the image patch.

[0245] In an exemplary embodiment, as shown in FIG9, when correcting the initial transformation parameters based on the control parameters and the frequency equalization matrix, the initial transformation parameters can first be corrected based on the frequency equalization matrix to obtain the first transformation coefficient; then, the correction coefficient is obtained based on the preset matrix and the control parameters; after that, the first transformation coefficient is corrected based on the correction coefficient to obtain the corrected second transformation coefficient.

[0246] The first correction to the initial transformation parameters based on the frequency equalization matrix can be achieved by performing a convolution operation on the frequency equalization matrix and the initial transformation parameters, as shown in Figure 9. The convolution operation uses... This means that the resulting value can be called the first transformation coefficient. For example, the first transformation coefficient can be obtained according to the following formula:

[0247] Where z represents the first transform coefficient, and C is the initial transform parameter, which is the transform coefficient calculated by the encoding standard of the existing encoder, with side lengths of 4, 8, 16 and 32.

[0248] The preset matrix I can be an identity matrix with the same dimension as the initial transformation coefficients C, which ensures that the control parameters correct the initial transformation coefficients C in the same dimension.

[0249] In obtaining the correction coefficient based on the preset matrix and control parameters, the preset matrix can be used as an increment to correct the control parameters, thereby obtaining the correction coefficient. As shown in Figure 8, in an exemplary embodiment, when obtaining the correction coefficient, a preset first coefficient β and a preset second coefficient α can be introduced. The preset first coefficient can be used to perform nonlinear mapping on the control parameters to ensure a smoother transition region between low-frequency and high-frequency components in the image block, thereby avoiding the introduction of obvious boundary effects in the image block. Furthermore, the preset second coefficient α is multiplied by the preset matrix, and the result of the nonlinear mapping and the product of the preset second coefficient α and the preset matrix are used as the correction coefficient.

[0250] For example, nonlinear mapping can be performed in the following ways. For instance, the control parameter can be nonlinearly mapped based on a preset first coefficient and the maximum threshold corresponding to the control parameter. This can be achieved by using the maximum threshold as the denominator and the control parameter as the numerator, performing an exponential operation on their values, and using the preset first coefficient as the weight.

[0251] In one embodiment, a third coefficient can be introduced to perform an exponential operation on the ratio between the maximum threshold and the control parameter. Specifically, a nonlinear mapping can be performed according to the following formula (5).

[0252] In formula (3), F' represents the result of nonlinear mapping, F represents the control parameter; MAX(F) represents the maximum threshold corresponding to the control parameter, and γ represents the third coefficient.

[0253] Here, MAX(F) refers to the maximum value among all the control parameters corresponding to all image blocks in the current frame. For example, if the control parameters are distributed between 0 and 4, then MAX(F) can be the maximum value among 0 and 4.

[0254] As can be seen, the first coefficient is preset as the weight of the nonlinear mapping, and the third coefficient is preset as the exponent of the nonlinear mapping. The third coefficient can be preset based on empirical values, and the steepness of the transformation can be controlled by the γ parameter. The larger γ is, the more significant the influence of the control parameter F on the equalization process; conversely, the transformation is smoother, and the influence of the control parameter F is relatively weaker. Thus, the coefficient γ can be used to adjust the degree of influence of the control parameter F on the equalization process.

[0255] Of course, in some examples, the third coefficient can also be determined based on the type of the current frame in the video stream, which can characterize whether the current frame is an I-frame, P-frame, or B-frame. For example, the third coefficient corresponding to the current frame being an I-frame can be greater than the third coefficient corresponding to P-frames and B-frames. In this way, more image information of keyframes can be preserved, enhancing the influence of the control parameter F on I-frames.

[0256] For example, the result of a preset second coefficient α and a preset matrix can be obtained in the following way, and this result can be used as an increment to correct the result of the nonlinear mapping.

[0257] For example, a correction value for correcting the nonlinear mapping can be obtained based on a preset matrix and a preset second coefficient. Specifically, it can be done according to the following formula (6): x=α·I Formula (6);

[0258] Where I is a preset matrix, · represents multiplication, and x represents a correction value, which provides a basic adjustment amount that can be the same for all image patches.

[0259] Next, the correction coefficients can be obtained based on the correction value x and the result F' of the nonlinear mapping.

[0260] For example, the correction factor can be obtained according to the following formula (7):

[0261] Where y represents the correction coefficient.

[0262] After obtaining the correction coefficients, the first transformation coefficients need to be corrected. For example, the first transformation coefficients can be corrected according to the following formula (8) to obtain the second transformation coefficients, which are the corrected initial transformation parameters:

[0263] Where C' is the second transformation coefficient, which is the corrected initial transformation parameter.

[0264] In formula (8), ⊙ represents element-wise multiplication. This represents convolution.

[0265] In this embodiment, the values ​​of the preset first coefficient and the preset second coefficient are both in the range of 0 to 1.

[0266] By using the above process, the corrected initial transformation parameters can be obtained. Then, based on the initial transformation parameters, the transformation of each image block can be completed, thereby adaptively transforming the image block from the spatial domain to the frequency domain.

[0267] Next, we can proceed to step S402, where each image block converted to the frequency domain is sequentially quantized and entropy encoded to obtain a precoded frame.

[0268] Quantization can selectively remove information (lossy parts), or more simply, it can compress image patches based on quantization coefficients. In practice, uniform quantization can be used to achieve this.

[0269] Entropy coding can employ arithmetic coding or VLC coding to compress the current frame.

[0270] Thus, the encoding of the current frame is completed, and the encoded frame is called a precoded frame.

[0271] Next, in step S403, a video quality distortion model can be constructed based on the precoded frame and the control parameters. This video quality distortion model can be used to evaluate the image quality after encoding the current frame. For example, it can be used to evaluate the difference between the precoded frame and the current frame, which reflects the degree of distortion of the precoded frame compared to the current frame.

[0272] Among them, the control parameters can be involved in the construction of the video quality distortion model. Specifically, the control parameters can reflect the importance of image patches to visual attention. Thus, the video quality distortion model constructed by the control parameters can reflect the degree of distortion of image regions with different visual attention, thereby guiding the coding to ensure the fidelity of image regions with high visual attention and avoid the distortion of image regions with high visual attention.

[0273] In step S404, the difference between the current frame and the pre-coded frame can be obtained based on the video quality distortion model, thereby correcting the encoding process of the current frame.

[0274] In an exemplary embodiment, the method of image segmentation of the current frame can be modified based on a video quality distortion model, and the size of each image block can be changed. In this new image segmentation method, the parameters of the re-segmented image blocks can be determined, frequency transformed, quantized and entropy encoded in sequence.

[0275] In an exemplary embodiment, since video transmission quality is affected by network bandwidth, bit error rate and packet loss rate in the channel, in order to adapt to the network environment and balance the clarity of areas of high visual attention and video transmission efficiency, network parameters can be introduced when constructing a video quality distortion model. The video quality distortion model can be constructed by using network parameters, control parameters and precoded frames.

[0276] The network parameters may include the network bandwidth of the networks where the sending and receiving ends are located, the bit error rate and packet loss rate of the communication channel between the sending and receiving ends.

[0277] Among them, bandwidth can affect the real-time performance of video transmission, bit error rate and packet loss rate are related to the degree of distortion of the video received by the receiver, and control parameters are related to the visual attention of the image, that is, related to subsequent video analysis. Therefore, the video quality distortion model constructed based on network parameters, control parameters and precoded frames can characterize the impact of visual attention, network bandwidth, bit error rate and packet loss rate of different image regions in the current frame on video transmission quality.

[0278] Here, the communication channel refers to the communication channel between the sending end and the receiving end, and the network bandwidth can refer to the bandwidth of the network where the sending end and the receiving end are located.

[0279] In some examples, there can be multiple communication channels between the sender and receiver, such as Bluetooth, broadband, and 5G networks. In this case, a video quality distortion model can be established for each communication channel to evaluate the distortion of video transmission in each channel. This allows for the selection of a communication channel that balances video analysis and transmission efficiency as the final video transmission channel to complete video encoding and transmission.

[0280] In this context, the sending end refers to the device that encodes the video stream, and the receiving end refers to the device that receives the encoded video stream.

[0281] Referring to Figure 10, which illustrates the process of constructing a video quality distortion model, the network feature parameters in Figure 10 are the network parameters of this embodiment. The video feature parameters include a first parameter and a second parameter. The encoder refers to the encoder that performs video encoding in this embodiment. The channel refers to the communication channel between the transmitting end and the receiving end. The binocular video signal source is used to provide a texture map, and the depth image video signal source is used to provide a depth map. The process can be as follows:

[0282] First, based on the precoded frame and the current frame, obtain the distortion value corresponding to the current frame;

[0283] Next, a video quality distortion model is constructed based on the control parameters, network parameters, and distortion values;

[0284] The distortion value is used to characterize the difference between the precoded frame and the current frame.

[0285] In this embodiment, the distortion value corresponding to the current frame can be obtained based on the difference between the precoded frame and the current frame. This distortion value is used to evaluate the degree of encoding distortion of the current frame in this encoding.

[0286] The video quality distortion model can be constructed according to the following formula (9): G=f(p net (b,e,l),p videoFormula (9) (L,F))+D1

[0287] Where G represents the video quality distortion model, b represents the network channel bandwidth, e represents the bit error rate, and l represents the network packet loss rate; p video The parameters represent the control parameters, with L being the first parameter, F being the second parameter, and D1 representing the distortion value.

[0288] In one example of this embodiment, since each frame in the video stream includes depth information and texture information, when obtaining the distortion value, the degree of encoding distortion of the current frame can be evaluated in both the image depth and image texture dimensions, thereby improving the precision of constructing the video quality distortion model.

[0289] Specifically, when obtaining the distortion value, the depth map and texture map of the current frame can be obtained. Then, the first encoded map corresponding to the texture map in the pre-coded frame can be obtained. The first encoded map can be understood as being obtained by performing the above encoding on the texture map. Then, the second encoded map corresponding to the depth map can be obtained. The second encoded map can be understood as being obtained by performing the above encoding on the depth map.

[0290] The first distortion value can be obtained based on the difference between the first encoded map and the texture map. This first distortion value can characterize the degree of texture distortion when encoding the current frame.

[0291] The second distortion value can be obtained based on the difference between the second coding map and the depth map. This second distortion value can characterize the degree of depth distortion when encoding the current frame.

[0292] In practice, the sum or average of the first and second distortion values ​​can be used as the distortion value corresponding to the current frame.

[0293] In this embodiment, the first distortion value, the second distortion value, and the distortion value can be values ​​in the range of 0-1.

[0294] Accordingly, when constructing a video quality distortion model based on distortion values, network parameters, and control parameters, it can be constructed according to the following formula (10): G = f(p net (b,e,l),p video (L,F))+D t +D d Formula (10);

[0295] Among them, D t D represents the first distortion value. d This represents the second distortion value, i.e., D1 = D mentioned above. t +D d .

[0296] In an exemplary embodiment, a rate-distortion model is typically constructed during video encoding. This rate-distortion model is mainly used to optimize compression efficiency during encoding, achieving the best encoding effect by balancing the compression ratio and distortion.

[0297] After constructing the video quality distortion model, the encoder's original rate-distortion model can be corrected based on the video quality distortion model and the output bitrate of each image block in the pre-coded frame. Then, the current frame can be re-encoded based on the corrected rate-distortion model.

[0298] The coding parameters used for encoding can be obtained based on the modified rate-distortion model. These coding parameters may include quantization coefficients and image patch size parameters, and thus the current frame can be re-encoded according to the new coding parameters.

[0299] When re-encoding the current frame, the process of steps S100-S400 described above is still followed.

[0300] Therefore, new video quality distortion models can be continuously constructed to obtain new coding parameters, thereby obtaining pre-coded frames after encoding the current frame each time. In practice, the video quality distortion model can be used to measure the video distortion. In practice, the pre-coded frame corresponding to the output bitrate within the preset bias range can be determined as the final coded image for output. In this way, smaller distortion and higher transmission efficiency can be achieved, thus taking into account both the quality of video analysis and communication channels.

[0301] It should be noted that the control parameters in this embodiment can be applied in the stage of frequency conversion of image blocks and in the construction of video quality distortion models. As described in the above embodiments, the control parameters may include a first parameter and a second parameter. The first parameter may represent the priority of the image block to visual attention, and the second parameter may represent the importance of the image block to visual attention.

[0302] In one example, the second parameter can be applied during the frequency conversion stage of the image patch to adjust the high-frequency components. The second and first parameters can be used together to construct the video quality distortion model. This allows the second parameter to specifically characterize the visual attention of the image patch, ensuring that the frequency conversion preserves image details in areas of high visual attention, thus maintaining sharpness. The first and second parameters, on the other hand, allow for a more refined video quality distortion model.

[0303] Thus, in the above formulas (8) and (7), F represents the second parameter.

[0304] In the above formulas (9) and (10), the control parameters include the first parameter L and the second parameter F.

[0305] The video encoding method of this disclosure embodiment will now be described with reference to a specific example. Please refer to Figure 11, which shows a flowchart of the specific vector video encoding method.

[0306] Step 1: Acquire a real-time video stream containing texture and depth information using a binocular camera. This video stream has stereo vision.

[0307] Step 2: Use the trained target network to obtain the control parameters of each image block in the current frame, thereby obtaining the visual attention information of each image block.

[0308] It should be noted that this part of the network was trained according to Pre_Step1 to Pre_Step3.

[0309] Pre_Step 1: Prepare the dataset, which includes multiple video samples. Each pixel in each frame of each video sample carries a control parameter label to achieve self-supervised training.

[0310] As shown in Figure 11, in this step, visual features of people that conform to video coding characteristics can be defined through annotation.

[0311] Pre_Step 2: Combine the video analysis to be performed, perform automatic annotation, and build a dataset;

[0312] Pre_Step 3: Self-supervised model training, i.e., training the labeled network model in Figure 11.

[0313] Step 3: Based on the adjustment parameters (first parameters) of each image block obtained in Step 2, the initial transformation parameters (i.e., the transformation coefficients in Figure 11) to be used for frequency conversion are adaptively adjusted. The adjustment process can be carried out with reference to formula (7) to optimize the details of areas with high visual attention.

[0314] Step 4: According to the above formula (9), establish a video quality distortion model based on network parameters and control parameters (first parameter and second parameter), then optimize the rate distortion model, and re-encode the current frame based on the encoding parameters output by the rate distortion model to ensure the smoothness and clarity of video encoding under limited bandwidth conditions.

[0315] The video content features in Figure 11 are the first and second parameters, and the network features are the network parameters mentioned above.

[0316] It should be noted that the video encoding method described above can be used to generate 3D video streams, i.e., three-dimensional video streams.

[0317] Based on the same inventive concept, this disclosure also provides a video encoder, which may include a processing module configured to encode each frame in a video stream to be encoded, the video stream being acquired by a binocular image acquisition device.

[0318] Specifically, the processing module is configured to, when encoding the current frame in the video stream, obtain the motion vectors and depth values ​​corresponding to multiple image blocks in the current frame, and obtain the control parameters corresponding to each of the multiple image blocks based on the depth values ​​and motion vectors; and encode the current frame based on the control parameters corresponding to each of the multiple image blocks.

[0319] Among them, the depth value represents the depth of field of the image patch in the current frame, the motion vector represents the motion of the image patch in the video stream, and the control parameter is used to represent the visual attention of the image patch in the video stream and to participate in the discarding of redundant information in the current frame during encoding.

[0320] In one exemplary embodiment, the processing module may include:

[0321] The first processing unit is configured to obtain the first attention level of the image patch based on the depth value;

[0322] The second processing unit is configured to obtain the second attention level of the image patch based on the motion vector;

[0323] The third processing unit is configured to acquire control parameters based on the first level of attention and the second level of attention.

[0324] In one exemplary embodiment, the second processing unit is specifically configured to: obtain at least three depth value ranges corresponding to the current frame based on the maximum depth value of the current frame, and obtain a second attention level based on the depth value range in which the depth value is located among the multiple depth value ranges;

[0325] Different depth value ranges correspond to different levels of secondary attention.

[0326] In one exemplary embodiment, the processing module may include:

[0327] The conversion unit is configured to convert each image block based on the control parameters corresponding to each image block and the initial transformation parameters corresponding to each image block; wherein, the conversion is used to convert the image block from the spatial domain to the frequency domain.

[0328] The quantization unit is configured to quantize the transformed image patches;

[0329] Entropy coding unit is configured to entropy code quantized image blocks to obtain precoded frames;

[0330] The video encoder also includes:

[0331] The correction module is configured to construct a video quality distortion model for the current frame based on the pre-encoded frame and control parameters, and to correct the encoding performed by the processing module based on the video quality distortion model.

[0332] In one exemplary embodiment, the control parameters include a first parameter and a second parameter, which are obtained by performing different calculations on a first attention level and a second attention level.

[0333] The transformation unit is specifically configured to adjust the initial transformation parameters corresponding to each image block based on the first parameter corresponding to each image block;

[0334] The correction module is specifically configured to construct a video quality distortion model for the current frame based on the precoded frame, the first parameter, and the second parameter.

[0335] In one exemplary embodiment, the video encoder may further include:

[0336] The input module is configured to input network parameters to the processing module. The network parameters include the network bandwidth of the network where the video encoder is currently located, and the bit error rate and packet loss rate of the various communication channels between the video encoder and the receiver.

[0337] The correction module is specifically configured to construct a video quality distortion model based on precoded frames, control parameters, and network parameters.

[0338] In one exemplary embodiment, the correction module is specifically configured to: obtain the distortion value corresponding to the current frame based on the precoded frame and the current frame, and construct a video quality distortion model based on the control parameters, network parameters and the distortion value;

[0339] The distortion value is used to characterize the difference between the precoded frame and the current frame.

[0340] In one exemplary embodiment, the processing module further includes:

[0341] The image processing unit is configured to acquire the depth map and texture map of the current frame;

[0342] The prediction unit is configured to obtain depth values ​​based on the depth map and motion vectors based on the texture map;

[0343] Specifically, the correction unit is configured to acquire a first coded map corresponding to the texture map and a second coded map corresponding to the depth map in the precoded frame, and acquire a first distortion value based on the first coded map and the texture map; acquire a second distortion value based on the second coded map and the depth map; and acquire a distortion value based on the first distortion value and the second distortion value.

[0344] In one exemplary embodiment, a rate-distortion model is configured in the video encoder;

[0345] The correction module is specifically configured to correct the rate distortion model based on the video quality distortion model and the output bitrate of each image block in the precoded frame.

[0346] The processing module is specifically configured to re-encode the current frame based on the modified rate-distortion model.

[0347] In one exemplary embodiment, a target network is configured within the video encoder; the processing module is specifically configured to: input the current frame into the target network, and receive the control parameters corresponding to each image block in the current frame output by the target network;

[0348] The target network is obtained by self-supervised training of a preset network using multiple video samples as training samples. Each pixel in each frame of the video sample carries a control parameter label, which is obtained based on the motion vector and depth value of each predicted image patch. The control parameter label is used to participate in self-supervision during training. The predicted image patch is obtained by image segmentation of each frame of the video sample.

[0349] In one exemplary embodiment, the target network includes:

[0350] The first feature extraction module is used to extract features from the texture map of the current frame to obtain texture features;

[0351] The second feature extraction module is used to extract features from the depth map of the current frame to obtain depth features;

[0352] The fusion module is used to fuse texture features and depth features to obtain fused features;

[0353] The prediction module is used to obtain the control parameters of each pixel in the current frame based on the fused features;

[0354] The control parameters of the image block are obtained based on the control parameters of each pixel in the image block.

[0355] In one exemplary embodiment, the conversion unit is specifically configured to acquire the frequency equalization matrix corresponding to each image block; and to correct the initial transformation parameters based on the control parameters of the image block and the frequency equalization matrix, and to complete the conversion of each image block based on the corrected initial transformation parameters.

[0356] In one exemplary embodiment, when the conversion unit corrects the initial transformation parameters based on the image patch modulation parameters and the frequency equalization matrix, it is specifically configured as follows:

[0357] Based on the frequency equalization matrix, the initial transformation coefficients are first corrected to obtain the first transformation coefficients; and based on the preset matrix and the first parameter, the correction coefficients are obtained; and based on the correction coefficients, the first transformation coefficients are corrected to obtain the corrected second transformation coefficients.

[0358] In an exemplary embodiment, when the conversion unit obtains the correction coefficients based on a preset matrix and a first parameter, it is specifically configured as follows:

[0359] Based on the preset first coefficient and the maximum threshold corresponding to the first parameter, a nonlinear mapping is performed on the first parameter; and based on the preset matrix and the preset second coefficient, a correction value is obtained to correct the nonlinear mapping; and based on the correction value and the result of the nonlinear mapping, a correction coefficient is obtained.

[0360] Among them, the preset first coefficient and the preset second coefficient are both greater than 0 and less than 1.

[0361] In an exemplary embodiment, when the transformation unit obtains the correction value for correcting the nonlinear mapping based on a preset matrix and preset second coefficients, it is specifically configured as follows:

[0362] Obtain the third coefficient corresponding to the nonlinear mapping, and perform a nonlinear mapping on the first parameter based on the maximum threshold, with the preset first coefficient as the weight and the third coefficient as the exponent.

[0363] In one exemplary embodiment, when the conversion unit obtains the third coefficient corresponding to the nonlinear mapping, it is specifically configured to obtain the third coefficient based on the type of the current frame in the video stream.

[0364] The relevant aspects of the video encoder embodiment can be referred to the video encoding method embodiment described above, and will not be repeated here.

[0365] In one example, the video encoder described above can be a 3D-HEVC encoder. The video encoding method described above can be applied to the 3D-HEVC encoder to encode the video image at each viewpoint, thereby reducing redundant information while ensuring the clarity of the 3D video in areas of high visual attention.

[0366] This disclosure also provides an electronic device including a processor that executes the content described in the video encoding method embodiments during runtime.

[0367] This disclosure also provides a non-transitory computer-readable storage medium storing content thereon that, when executed by a computer's processor, causes the processor to implement the content described in the video encoding method embodiments.

[0368] It should be noted that the video stream encoded using the video encoding method or video encoder provided in this disclosure can be decoded using the video decoder corresponding to the video encoder. The decoding standard corresponds to the original video encoder. For example, if the encoder is the H.265 / HEVC standard, the decoder is also the H.265 / HEVC standard. Therefore, even if the encoding is improved based on the control parameters on the encoder side, the decoder can still decode correctly. Thus, the video encoder and video encoding method of this embodiment can be universally applied to devices.

[0369] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0370] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0371] The video encoding method, video encoder, electronic device, and medium provided in this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this disclosure. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

[0372] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0373] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

[0374] The terms "an embodiment," "embodiment," or "one or more embodiments" as used herein mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of this disclosure. Furthermore, please note that the examples of the phrase "in one embodiment" do not necessarily all refer to the same embodiment.

[0375] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this disclosure may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0376] In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This disclosure can be implemented by means of hardware comprising a plurality of different elements and by means of a suitably programmed computer. In a unit claim enumerating a plurality of means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0377] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit them. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure.

Claims

1. A video encoding method, wherein, The method includes: Obtain the video stream to be encoded; For the current frame in the video stream, obtain the motion vector and depth value corresponding to each of the multiple image blocks in the current frame; wherein, the depth value represents the distance between the content represented by the image block and the video acquisition device, and the motion vector represents the movement of the image block in the video stream; Based on the depth value and the motion vector, obtain the control parameters corresponding to each of the multiple image blocks; The current frame is encoded based on the control parameters corresponding to each of the multiple image blocks; The control parameters are used to characterize the visual attention of the image block in the video stream and at least to participate in the discarding of redundant information in the current frame during the encoding.

2. The video encoding method according to claim 1, wherein, The encoding of the current frame based on the control parameters corresponding to each of the multiple image blocks includes: Based on the control parameters corresponding to each image block and the initial transformation parameters corresponding to each image block, each image block is transformed; wherein, the transformation is used to transform the image block from the spatial domain to the frequency domain; Each of the converted image blocks is sequentially quantized and entropy encoded to obtain a precoded frame; Based on the precoded frames and the control parameters, a video quality distortion model is constructed; Based on the video quality distortion model, the encoding of the current frame is corrected.

3. The video encoding method according to claim 2, wherein, The control parameters include a first parameter and a second parameter, which are obtained by performing different calculations on the motion vector and the depth value. The adjustment parameters and initial transformation parameters corresponding to each image block include: Based on the first parameter corresponding to each image block, the initial transformation parameters corresponding to the image block are adjusted; The step of constructing a video quality distortion model based on the precoded frame and the control parameters includes: Based on the precoded frame, the first parameter, and the second parameter, a video quality distortion model for the current frame is constructed.

4. The video encoding method according to any one of claims 1-3, wherein, The step of obtaining the control parameters corresponding to each of the multiple image patches based on the depth value and the motion vector includes: Based on the depth value, the first attention level of the image patch is obtained; Based on the motion vector, the second attention level of the image patch is obtained; The control parameters are obtained based on the first level of attention and the second level of attention; Both the first attention level and the second attention level characterize the degree to which the image patch attracts visual attention.

5. The video encoding method according to claim 4, wherein, The step of obtaining the second attention level of the image patch based on the motion vector includes: Based on the maximum depth value of the current frame, obtain at least three depth value ranges corresponding to the current frame; The second attention level is obtained based on the depth value range in which the depth value falls within the multiple depth value ranges; Different depth value ranges correspond to different second attention levels.

6. The video encoding method according to any one of claims 2-3 or 5, wherein, The method further includes: obtaining network parameters, the network parameters including the network bandwidth of the networks where the sending end and the receiving end are located, the bit error rate and packet loss rate of the communication channel between the sending end and the receiving end; The step of constructing a video quality distortion model based on the precoded frame and the control parameters includes: The video quality distortion model is constructed based on the precoded frame, the control parameters, and the network parameters.

7. The video encoding method according to claim 6, wherein, The step of constructing the video quality distortion model based on the precoded frame, the modulation parameters, and the network parameters includes: Based on the precoded frame and the current frame, obtain the distortion value corresponding to the current frame; Based on the control parameters, the network parameters, and the distortion value, the video quality distortion model is constructed. The distortion value is used to characterize the difference between the precoded frame and the current frame.

8. The video encoding method according to claim 7, wherein, The step of obtaining the distortion value corresponding to the current frame based on the precoded frame and the current frame includes: Obtain the depth map and texture map of the current frame; wherein the depth value is obtained based on the depth map, and the motion vector is obtained based on the texture map; Obtain the first encoded map corresponding to the texture map and the second encoded map corresponding to the depth map in the precoded frame; Based on the first encoded image and the texture image, a first distortion value is obtained; Based on the second encoded map and the depth map, a second distortion value is obtained; The distortion value is obtained based on the first distortion value and the second distortion value.

9. The video encoding method according to claim 4, wherein, The step of correcting the encoding of the current frame based on the video quality distortion model includes: Based on the video quality distortion model and the output bitrate of each image block in the precoded frame, the initial rate distortion model is corrected; Based on the corrected rate-distortion model, the current frame is re-encoded to correct the encoding of the current frame.

10. The video encoding method according to any one of claims 1-3 or 5, wherein, The step of obtaining the control parameters corresponding to each of the multiple image patches based on the depth value and the motion vector includes: Input the current frame into the target network; Receive the control parameters corresponding to each image block in the current frame output by the target network; The target network is obtained by self-supervised training of a preset network using multiple video samples as training samples. Each frame of the video sample carries multiple pixels with adjustment parameter labels. The adjustment parameter labels are obtained based on the motion vector and depth value of each predicted image block. The adjustment parameter labels are used to participate in the self-supervision during the training process. The predicted image block is obtained by image segmentation of each frame of the video sample.

11. The video encoding method according to claim 10, wherein, The target network includes: The first feature extraction module is used to extract features from the texture map of the current frame to obtain texture features; The second feature extraction module is used to extract features from the depth map of the current frame to obtain depth features; The fusion module is used to fuse the texture features and the depth features to obtain fused features; The prediction module is used to obtain the adjustment parameters of each pixel in the current frame based on the fused features; The control parameters corresponding to the image block are obtained based on the control parameters of each pixel within the image block.

12. The video encoding method according to claim 10, wherein, The target network is obtained through the following steps: Obtain the depth map and texture map of each frame of the video sample; The depth map and the texture map are input into the preset network to obtain the prediction control parameters corresponding to multiple pixels of each frame of the video sample output by the preset network; Based on the control parameter labels corresponding to multiple pixels and the predicted control parameters, the parameters of the preset network are updated multiple times to obtain the target network.

13. The video encoding method according to any one of claims 2-3 or 7-12, wherein, The transformation of each image block based on the control parameters and the initial transformation parameters corresponding to each image block includes: Obtain the frequency equalization matrix corresponding to each image block; The initial transformation parameters are corrected based on the control parameters of the image patch and the frequency equalization matrix. The transformation of each image block is completed based on the corrected initial transformation parameters.

14. The video encoding method according to claim 13, wherein, The modification of the initial transformation parameters based on the adjustment parameters of the image patch and the frequency equalization matrix includes: Based on the frequency equalization matrix, the initial transformation parameters are corrected to obtain the first transformation coefficients; Based on the preset matrix and the control parameters, the correction coefficient is obtained; Based on the correction coefficient, the first transformation coefficient is corrected to obtain the corrected second transformation coefficient.

15. The video encoding method according to claim 14, wherein, When obtaining the correction coefficient based on the preset matrix and the control parameters, the process includes: Based on the preset first coefficient and the maximum threshold corresponding to the control parameter, the control parameter is nonlinearly mapped; Based on the preset matrix and the preset second coefficient, obtain the correction value for correcting the nonlinear mapping; The correction coefficient is obtained based on the correction value and the result of the nonlinear mapping; Wherein, both the preset first coefficient and the preset second coefficient are greater than 0 and less than 1.

16. The video encoding method according to claim 15, wherein, When obtaining the correction value for correcting the nonlinear mapping based on a preset matrix and a preset second coefficient, the process includes: Obtain the third coefficient corresponding to the nonlinear mapping; Based on the maximum threshold, and using the preset first coefficient as the weight and the third coefficient as the exponent, the first parameter is nonlinearly mapped.

17. The video encoding method according to claim 16, wherein, Obtaining the third coefficient corresponding to the nonlinear mapping includes: The third coefficient is obtained based on the type of the current frame in the video stream.

18. The video encoding method according to claim 1, wherein, The encoding of the current frame based on the control parameters corresponding to each of the multiple image blocks includes: Based on the control parameters corresponding to each of the multiple image blocks, each viewpoint image of the current frame is encoded to encode the video stream into a three-dimensional video stream; The multi-viewpoint image of the current frame is an image composed of multiple viewpoint images with parallax from the same perspective.

19. A video encoder, wherein, Includes a processing module configured to encode each frame in the video stream to be encoded; The processing module is specifically configured to execute the video encoding method according to any one of claims 1-18.

20. An electronic device, wherein, The electronic device includes a processor that, when running, executes the video encoding method according to any one of claims 1-18.

21. A non-transitory computer-readable storage medium, wherein, It stores, when executed by the computer's processor, the video encoding method as described in any one of claims 1-18.

Images