Encoding method, decoding method, encoders, decoders and storage medium

Abstract

The present application provides an encoding method, a decoding method, encoders, decoders and a storage medium. The decoding method comprises: determining one or more pieces of first indication information in a code stream, the first indication information being used for indicating first units; determining one or more first units in the code stream on the basis of the one or more pieces of first indication information in the code stream; determining one or more second units among the first units; parsing the second unit to determine a parsing result of the second unit; and on the basis of the parsing result of the second unit, obtaining a reconstructed image.

Description

An encoding / decoding method, encoder, decoder, and storage medium Technical Field

[0001] This application relates to the field of video encoding and decoding technology, and in particular to an encoding and decoding method, encoder, decoder and storage medium. Background Technology

[0002] Video encoding and decoding technology organizes video streams in the form of one or more sub-streams. In some scenarios, the encoded data in the sub-streams is encapsulated in the form of Network Abstraction Layer (NAL) units. The data units of the video stream and the sub-streams are distinguished by the size of the data packets identified in the stream. This requires the decoder or system layer to start scanning from the beginning of the video stream or sub-stream when accessing one of the data packets, because the beginning position of each data packet is calculated by adding up the sizes of the preceding data packets.

[0003] However, when packet loss or errors occur during the transmission of video streams and sub-streams, it is impossible to quickly determine the next correct data packet, affecting encoding and decoding performance. Summary of the Invention

[0004] This application provides an encoding / decoding method, an encoder, a decoder, and a storage medium.

[0005] In a first aspect, this application provides a decoding method applied to a decoder, the method comprising:

[0006] Determine one or more first indication information in the bitstream, the first indication information being used to indicate the first unit;

[0007] Based on one or more first indication information in the bitstream, determine one or more first units in the bitstream;

[0008] Identify one or more second units within the first unit;

[0009] Analyze the second unit to determine the results of the analysis;

[0010] Based on the analysis results of the second unit, the reconstructed image is obtained.

[0011] Secondly, embodiments of this application provide an encoding method applied to an encoder, the method comprising:

[0012] Preprocessing operations are performed on the original image to obtain the information to be encoded;

[0013] The information to be encoded is encoded to obtain one or more second units;

[0014] Based on the one or more second units, a first unit is generated;

[0015] Determine the first instruction information for the first unit;

[0016] A code stream is generated based on the first indication information of the first unit, whereby the first indication information is used to indicate the first unit.

[0017] Thirdly, embodiments of this application provide an encoder, which includes a first processing unit configured to perform preprocessing operations on the original image to obtain information to be encoded;

[0018] An encoding unit is configured to encode information to be encoded, resulting in one or more second units;

[0019] The encoding unit is further configured to generate a first unit based on the one or more second units;

[0020] The encoding unit is further configured to determine first indication information of the first unit; and to generate a code stream based on the first indication information of the first unit, wherein the first indication information is used to indicate the first unit.

[0021] Fourthly, embodiments of this application provide an encoder, which includes a first memory and a first processor; wherein,

[0022] A first memory for storing computer programs that can run on a first processor;

[0023] The first processor is used to execute the method described in the second aspect when running a computer program.

[0024] Fifthly, embodiments of this application provide a decoder, which includes a decoding unit and a second processing unit; wherein:

[0025] The decoding unit is configured to determine one or more first indication information in the bitstream, the first indication information being used to indicate a first unit; and to determine one or more first units in the bitstream based on one or more first indication information in the bitstream.

[0026] The decoding unit is also configured to determine one or more second units in the first unit;

[0027] The decoding unit is also configured to parse the second unit and determine the parsing result of the second unit;

[0028] The second processing unit is also configured to obtain the reconstructed image based on the analysis results of the second unit.

[0029] Sixthly, embodiments of this application provide a decoder, which includes a second memory and a second processor; wherein,

[0030] The second memory is used to store computer programs that can run on the second processor;

[0031] The second processor is used to execute the methods described in the first aspect when running a computer program.

[0032] In a seventh aspect, embodiments of this application provide a computer-readable storage medium that stores a bitstream generated by such encoding method.

[0033] Eighthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed, implements the method as described in the first aspect or the method as described in the second aspect.

[0034] This application provides an encoding / decoding method, encoder, decoder, and storage medium. For multi-layered data units, a first unit is quickly identified using first indication information, and sub-units within the first unit are further quickly identified using second indication information, thereby correctly decoding the data of the first unit and enabling random access to any first unit and its sub-units in the bitstream. Furthermore, when packet loss or bit errors are detected, the erroneous data unit can be discarded, and the next complete and error-free data packet can be scanned to begin correct decoding. Attached Figure Description

[0035] Figure 1 is a schematic block diagram of a video encoding and decoding system according to an embodiment of this application;

[0036] Figure 2 is a schematic block diagram of a video encoder involved in an embodiment of this application;

[0037] Figure 3 is a schematic block diagram of a video decoder involved in an embodiment of this application;

[0038] Figure 4 shows a schematic diagram of the network architecture of an encoding / decoding system provided in an embodiment of this application;

[0039] Figure 5 is a schematic diagram of the V3C stream structure provided in an embodiment of this application;

[0040] Figure 6 is a schematic flowchart of a decoding method provided in an embodiment of this application;

[0041] Figure 7 is a schematic diagram of a code stream structure in an embodiment of this application;

[0042] Figure 8 is a schematic diagram of a VCC stream structure in an embodiment of this application;

[0043] Figure 9 is a schematic diagram of another VCC stream structure in an embodiment of this application;

[0044] Figure 10 is a detailed structural diagram of a VCM stream structure in an embodiment of this application;

[0045] Figure 11 is a detailed structural diagram of another VCM stream structure in an embodiment of this application;

[0046] Figure 12 is a schematic flowchart of an encoding method provided in an embodiment of this application;

[0047] Figure 13 is a schematic diagram of the composition structure of an encoder provided in an embodiment of this application;

[0048] Figure 14 is a schematic diagram of the specific hardware structure of an encoder provided in an embodiment of this application;

[0049] Figure 15 is a schematic diagram of the composition structure of a decoder provided in an embodiment of this application;

[0050] Figure 16 is a schematic diagram of the specific hardware structure of a decoder provided in an embodiment of this application;

[0051] Figure 17 is a schematic diagram of the composition structure of an encoding / decoding system provided in an embodiment of this application. Detailed Implementation

[0052] This application can be applied to the fields of image encoding and decoding, video encoding and decoding, hardware video encoding and decoding, dedicated circuit video encoding and decoding, and real-time video encoding and decoding. For example, the solution of this application can be combined with audio video coding standards (AVS), such as H.264 / Audio Video Coding (AVC), H.265 / High Efficiency Video Coding (HEVC), H.266 / Versatile Video Coding (VVC), Visual Volumetric Video-based Coding (V3C), Video Coding for Machine (VCM), and Feature Coding for Machine (FCM). Alternatively, the solutions in this application can be incorporated into other proprietary or industry standards, including ITU-TH.261, ISO / IEC MPEG-1 Visual, ITU-TH.262 or ISO / IEC MPEG-2 Visual, ITU-TH.263, ISO / IEC MPEG-4 Visual, and ITU-TH.264 (also known as ISO / IEC MPEG-4 AVC), which include Scalable Video Codec (SVC) and Multi-View Video Codec (MVC) extensions. It should be understood that the technology in this application is not limited to any particular codec standard or technology.

[0053] The high-degree-of-freedom immersive coding system can be roughly divided into the following stages according to the task line: data acquisition, data organization and expression, data encoding and compression, data decoding and reconstruction, data synthesis and rendering, and finally presenting the target data to the user.

[0054] The encoding involved in the embodiments of this application is mainly video encoding and decoding. For ease of understanding, the video encoding and decoding system involved in the embodiments of this application will be introduced first with reference to Figure 1.

[0055] Figure 1 is a schematic block diagram of a video encoding and decoding system according to an embodiment of this application. It should be noted that Figure 1 is only an example, and the video encoding and decoding system of this application includes, but is not limited to, the one shown in Figure 1. As shown in Figure 1, the video encoding and decoding system 100 includes an encoding device 110 and a decoding device 120. The encoding device is used to encode (can be understood as compressing) video data to generate a bitstream, and transmits the bitstream to the decoding device. The decoding device decodes the bitstream generated by the encoding device to obtain decoded video data.

[0056] The encoding device 110 in this application embodiment can be understood as a device with video encoding function, and the decoding device 120 can be understood as a device with video decoding function. That is, the encoding device 110 and the decoding device 120 in this application embodiment include a wider range of devices, such as smartphones, desktop computers, mobile computing devices, laptops (e.g., laptop computers), tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video game consoles, in-vehicle computers, etc.

[0057] In some embodiments, encoding device 110 may transmit encoded video data (such as a bitstream) to decoding device 120 via channel 130. Channel 130 may include one or more media and / or means capable of transmitting encoded video data from encoding device 110 to decoding device 120.

[0058] In one example, channel 130 includes one or more communication media that enable encoding device 110 to transmit encoded video data directly to decoding device 120 in real time. In this example, encoding device 110 can modulate the encoded video data according to a communication standard and transmit the modulated video data to decoding device 120. The communication media includes wireless communication media, such as radio frequency spectrum; optionally, the communication media may also include wired communication media, such as one or more physical transmission lines.

[0059] In another example, channel 130 includes a storage medium that can store video data encoded by encoding device 110. The storage medium includes various local access data storage media, such as optical discs, DVDs, flash memory, etc. In this example, decoding device 120 can retrieve the encoded video data from the storage medium.

[0060] In another example, channel 130 may include a storage server that can store the video data encoded by encoding device 110. In this example, decoding device 120 can download the stored encoded video data from the storage server. Optionally, the storage server can store and transmit the encoded video data to decoding device 120, such as a web server (e.g., for a website), a file transfer protocol (FTP) server, etc.

[0061] In some embodiments, the encoding device 110 includes a video encoder 112 and an output interface 113. The output interface 113 may include a modulator / demodulator (modem) and / or a transmitter.

[0062] In some embodiments, the encoding device 110 may include a video source 111 in addition to the video encoder 112 and the input interface 113.

[0063] The video source 111 may include at least one of a video capture device (e.g., a video camera), a video archive, a video input interface, and a computer graphics system, wherein the video input interface is used to receive video data from a video content provider, and the computer graphics system is used to generate video data.

[0064] Video encoder 112 encodes video data from video source 111 to generate a bitstream. The video data may include one or more pictures or a sequence of pictures. The bitstream contains the encoding information of the pictures or picture sequences in the form of a bitstream. The encoding information may include encoded image data and associated data. The associated data may include a sequence parameter set (SPS), a picture parameter set (PPS), and other syntax element structures. The SPS may contain parameters applied to one or more sequences. The PPS may contain parameters applied to one or more pictures. A syntax element structure refers to a set of zero or more syntax elements arranged in a specified order within the bitstream.

[0065] The video encoder 112 transmits the encoded video data directly to the decoding device 120 via the output interface 113. The encoded video data can also be stored on a storage medium or a storage server for subsequent retrieval by the decoding device 120.

[0066] In some embodiments, the decoding device 120 includes an input interface 121 and a video decoder 122. In some embodiments, in addition to the input interface 121 and the video decoder 122, the decoding device 120 may also include a display device 123.

[0067] The input interface 121 includes a receiver and / or a modem. The input interface 121 can receive encoded video data through channel 130.

[0068] The video decoder 122 is used to decode the encoded video data to obtain the decoded video data, and transmit the decoded video data to the display device 123.

[0069] Display device 123 displays the decoded video data. Display device 123 may be integrated with decoding device 120 or external to decoding device 120. Display device 123 may include various display devices, such as liquid crystal display (LCD), plasma display, organic light-emitting diode (OLED) display, or other types of display devices.

[0070] Furthermore, Figure 1 is merely an example, and the technical solutions of the embodiments of this application are not limited to Figure 1. For example, the technology of this application can also be applied to one-sided video encoding or one-sided video decoding.

[0071] The video coding framework involved in the embodiments of this application is described below.

[0072] Referring to Figure 2, it shows a schematic block diagram of an encoder provided in an embodiment of this application. As shown in Figure 2, the encoder (specifically a "video encoder") 200 may include a transform and quantization unit 101, an intra-frame estimation unit 102, an intra-frame prediction unit 103, a motion compensation unit 104, a motion estimation unit 105, an inverse transform and inverse quantization unit 106, a filter control and analysis unit 107, a filtering unit 108, an encoding unit 109, and a decoded image buffer unit 140, etc. Among them, the filtering unit 108 can implement deblocking filtering and sample adaptive offset (SAO) filtering, and the encoding unit 109 can implement header information encoding and context-based adaptive binary arithmetic coding (CABAC).For the input original video signal, a video coding block can be obtained through the division of a Coding Tree Unit (CTU). Then, the residual pixel information obtained after intra-frame or inter-frame prediction is transformed for the video coding block by a transform and quantization unit 101, including transforming the residual information from the pixel domain to the transform domain and quantizing the obtained transform coefficients to further reduce the bit rate; an intra-frame estimation unit 102 and an intra-frame prediction unit 103 are used for intra-frame prediction of the video coding block; specifically, the intra-frame estimation unit 102 and the intra-frame prediction unit 103 are used to determine the intra-frame prediction mode to be used for encoding the video coding block; a motion compensation unit 104 and a motion estimation unit 105 are used to perform inter-frame predictive coding of the received video coding block with respect to one or more blocks in one or more reference frames to provide temporal prediction information; the motion estimation performed by the motion estimation unit 105 is a process of generating a motion vector, and the motion vector can estimate the motion of the video coding block, and then the motion compensation unit 104 performs motion compensation based on the motion vector determined by the motion estimation unit 105; after determining the intra-frame prediction mode, the intra-frame prediction unit 103 is further used to provide the selected intra-frame prediction data to an encoding unit 109, and the motion estimation unit 105 also sends the calculated and determined motion vector data to the encoding unit 109; in addition, an inverse transform and inverse quantization unit 106 is used for the reconstruction of the video coding block, reconstructing the residual block in the pixel domain, and the reconstructed residual block removes block effect artifacts through a filter control analysis unit 107 and a filtering unit 108, and then the reconstructed residual block is added to a predictive block in a frame of a decoded image buffer unit 140 to generate a reconstructed video coding block; the encoding unit 109 is used for encoding various coding parameters and the quantized transform coefficients. In the CABAC-based coding algorithm, the context can be based on adjacent coding blocks and can be used for encoding information indicating the determined intra-frame prediction mode, and outputs the bitstream of the video signal; while the decoded image buffer unit 140 is used for storing the reconstructed video coding blocks for prediction reference. As the video image coding progresses, new reconstructed video coding blocks will be continuously generated, and these reconstructed video coding blocks will be stored in the decoded image buffer unit 140.

[0073] Referring to FIG. 3, which shows a schematic block diagram of a decoder provided in an embodiment of the present application. As shown in FIG. 3, the decoder (specifically, a "video decoder") 200 includes a decoding unit 201, an inverse transform and inverse quantization unit 202, an intra prediction unit 203, a motion compensation unit 204, a filtering unit 205, a decoded image buffer unit 206, etc. Among them, the decoding unit 201 can implement header information decoding and CABAC decoding, and the filtering unit 205 can implement deblocking filtering and SAO filtering. After the input video signal is encoded through the process shown in FIG. 1, the bitstream of the video signal is output; this bitstream is input into the decoder 200, and first passes through the decoding unit 201 to obtain the decoded transform coefficients; the inverse transform and inverse quantization unit 202 processes the transform coefficients to generate a residual block in the pixel domain; the intra prediction unit 203 can be used to generate prediction data for the current video decoding block based on the determined intra prediction mode and data from previously decoded blocks in the current frame or picture; the motion compensation unit 204 determines the prediction information for the video decoding block by analyzing the motion vector and other associated syntax elements, and uses this prediction information to generate a predictive block for the video decoding block being decoded; by summing the residual block from the inverse transform and inverse quantization unit 202 and the corresponding predictive block generated by the intra prediction unit 203 or the motion compensation unit 204, a decoded video block is formed; the decoded video signal passes through the filtering unit 205 to remove block effect artifacts, which can improve the video quality; then the decoded video block is stored in the decoded image buffer unit 206, and the decoded image buffer unit 206 stores reference images for subsequent intra prediction or motion compensation, and is also used for the output of the video signal, that is, the original video signal is restored.

[0074] Furthermore, an embodiment of the present application also provides a network architecture of an encoding and decoding system including an encoder and a decoder. Among them, FIG. 4 shows a schematic diagram of a network architecture of an encoding and decoding system provided in an embodiment of the present application. As shown in FIG. 4, this network architecture includes one or more electronic devices 13 to 1N and a communication network 01. Among them, the electronic devices 13 to 1N can perform video interaction through the communication network 01. During the implementation process, the electronic devices can be various types of devices with video encoding and decoding functions. For example, the electronic devices can include smart phones, tablet computers, personal computers, personal digital assistants, navigators, digital telephones, video telephones, televisions, sensing devices, servers, etc., which are not specifically limited here. In addition, the decoder or encoder described in the embodiment of the present application can be the above-mentioned electronic devices.

[0075] It should be noted that the embodiment of the present application can be applied to the encoder, can also be applied to the decoder, and even can be applied to both the encoder and the decoder at the same time, but the embodiment of the present application does not make specific limitations.

[0076] The above is the basic process of a video codec under a block-based hybrid coding framework. With the development of technology, some modules or steps of this framework or process may be optimized. This application is applicable to the basic process of a video codec under this block-based hybrid coding framework, but is not limited to this framework and process.

[0077] In some related technologies of the above video coding and decoding system (such as VVC), the encoded data is encapsulated in the form of NAL units, and a 24-bit start prefix code (Start Code) prefix is used to distinguish NAL units in the bitstream, or the identification at the system layer is relied on to distinguish NAL units in the bitstream, so that fast scanning and random access of each NAL unit can be achieved.

[0078] However, this technology needs to define a type of NAL unit for each type of data packet. The advantage of this is that it enables the decoder or the system layer to quickly scan and identify the type of each NAL unit. However, its defect is that each new added NAL unit needs to occupy a NAL unit type number, and the NAL unit types it can support are limited, and it cannot effectively support the situation where there are multiple sub-bitstreams and each sub-bitstream contains its own unique NAL unit type. Moreover, since all NAL units need to be identified, it increases the burden on the scanning of the decoder or the system layer.

[0079] In some other related technologies of the above video coding and decoding system, the encoded video bitstream contains multiple parallel sub-bitstreams. For example, the V3C video bitstream contains an Atlas sub-bitstream, an Attribute Video sub-bitstream, a Geometry Video sub-bitstream, etc. Another example is that the VCM video bitstream contains a kernel video data sub-bitstream obtained by the kernel encoder in the VCM encoder, and a reconstruction data sub-bitstream generated by the VCM encoder based on the reconstruction information obtained from its pre-processing operations on the video. Another example is that the FCM video bitstream also contains a video data sub-bitstream and a reconstruction data sub-bitstream.

[0080] FIG. 5 is a schematic diagram of the V3C bitstream structure provided by an embodiment of this application. Among them, the V3C bitstream includes: The V3C parameter set (V3C_parameter_set()) of V3C_VPS may include ptl_profile_toolset_idc. When ptl_profile_toolset_idc is 128 / 129 / 130, it indicates that the current bitstream contains both VPCC extend and MIV main types of bitstreams.

[0081] The Atlas sequence parameter set (Atlas_sequence_parameter_set_rbsp()) in NAL_ASPS of the V3C_AD stitched sub-bitstream (Atlas_sub_bitstream()) can include asps_vpcc_extension_present_flag and asps_miv_extension_present_flag. When ptl_profile_toolset_idc is 128 / 129 / 130, asps_vpcc_extension_present_flag is true (i.e., 1), and asps_miv_extension_present_flag is also true (i.e., 1).

[0082] The ACL NAL unit type (ACL_NAL_unit_type) in V3C_AD's Atlas_sub_bitstream() includes hybrid stitching information. For example, the atlas tile data unit (atlas_tile_data_unit()) can include atdu_type_flag. If atdu_type_flag is yes (i.e., 1), it indicates that the current tile belongs to a point cloud tile; if atdu_type_flag is no (i.e., 0), it indicates that the current tile belongs to a multi-view video tile.

[0083] Furthermore, the sub-tack information data (patch_information_data) includes sub-tack data units (patch_data_unit). If atdu_type_flag is negative and asps_miv_extension_present_flag is positive, it indicates that the current sub-tack is implemented using a multi-view video coding standard. If atdu_type_flag is positive, it indicates that the current sub-tack is implemented using a point cloud video decoding standard.

[0084] The data in the V3C_AD splicing diagram sub-bit stream (also called sub-code stream) is also encapsulated in the form of NAL units. However, unlike existing technologies, the NAL units in the sub-code stream are not distinguished by the start prefix code, but by the packet size of the data in each NAL unit identified in the code stream. In this way, the decoder can still achieve fast scanning and random access to the data in the sub-code stream by scanning from the head of the sub-code stream.

[0085] The video sub-bitstreams include: V3C_GVD video sub-bitstream(), V3C_AVD video sub-bitstream(), V3C_CAD video sub-bitstream(), and V3C_PVD video sub-bitstream().

[0086] In addition, to distinguish different sub-stream data within the video stream, the sub-stream data is split and encapsulated in V3C units. Each V3C unit records the type of sub-stream data it carries. The V3C units in the video stream are distinguished based on the data packet size of each V3C unit identified within the stream.

[0087] However, by encapsulating data units in two layers, the data units of the video stream and sub-stream are distinguished by the size of the data packets identified within the stream. This requires the decoder or system layer to start scanning from the beginning of the video stream or sub-stream when accessing a data packet, since the starting position of each data packet is calculated by summing the sizes of the preceding data packets. This is feasible in scenarios where data is stored and transmitted as files, allowing the decoder or system layer to obtain the complete video stream and scan from its beginning. However, in real-time transmission scenarios, when packet loss or errors occur during video stream transmission, the decoder or system layer cannot discard erroneous data and scan to the next complete and error-free data packet to begin correct decoding because the size of each data packet cannot be obtained. This prevents the decoder from achieving random access functionality.

[0088] Based on this, embodiments of this application provide an encoding / decoding method, an encoder, a decoder, and a storage medium. For multi-layered data units, a first unit is quickly identified using first indication information, and sub-units within the first unit are further quickly identified using second indication information, thereby correctly decoding the data of the first unit and enabling random access to any first unit and its sub-units in the bitstream. Moreover, when packet loss or bit errors are detected, the erroneous data unit can be discarded, and the next complete and error-free data packet can be scanned to begin correct decoding.

[0089] The above-mentioned related technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application. The embodiments of this application include at least some of the following contents.

[0090] In one embodiment of this application, referring to FIG6, a flowchart of a decoding method provided by an embodiment of this application is shown. As shown in FIG6, the method may include:

[0091] S601: Determine one or more first indication information in the bitstream, the first indication information being used to indicate the first unit;

[0092] The bitstream can be the video bitstream obtained after the encoder performs preprocessing and encapsulation operations on the input video, or it can be a sub-bitstream within the video bitstream. For example, in the V3C coding standard, the video bitstream includes atlas sub-bitstreams, attribute video sub-bitstreams, and geometry video sub-bitstreams. As another example, the VCM video bitstream includes a kernel video data sub-bitstream obtained from the kernel encoder in the VCM encoder, and a reconstructed data sub-bitstream generated by the VCM encoder based on the reconstructed information obtained from its preprocessing operations on the video. Similarly, the FCM video bitstream also includes video data sub-bitstreams and reconstructed data sub-bitstreams.

[0093] The first unit can be a specific data packet or data unit in the bitstream. The first unit includes one or more second units, where the second unit is a sub-data packet or sub-data unit within the first unit.

[0094] Figure 7 is a schematic diagram of a bitstream structure according to an embodiment of this application. As shown in Figure 7, the bitstream includes one or more first units and first indication information for indicating the first units; the first unit includes one or more second units and second indication information for indicating the second units. In one example, the first unit includes two parts: header information and payload data. The payload data of the first unit includes one or more second units, and the second unit includes two parts: header information and payload data.

[0095] S602: Based on one or more first indication information in the bitstream, determine one or more first units in the bitstream;

[0096] The first indication information is used to indicate the starting position of the first unit and / or the data packet size.

[0097] In some embodiments, the first indication information is used to indicate the starting position of a first unit in the bitstream, so that the decoder can quickly identify the first unit and thus correctly parse the data of the first unit. For example, scanning the bitstream to determine one or more pieces of first indication information in the bitstream.

[0098] For example, the first indication information is a start code with a first number of bits. The start code is a special bit pattern used in video bitstreams. It is typically a pattern consisting of a specific number of zero bits followed by a 1 bit. This design helps the decoder quickly locate the start position of the first unit by scanning the start code of the first unit in the received bitstream. The start position of the first unit can be any bit following the start code. For example, the start code of the first unit might be 23 zero bits followed by a 1 bit, i.e., the first bit length is 24 bits, and the format could also be represented as 3 bytes of "00 00 01". As another example, the start code of the first unit might be 31 zero bits followed by a 1 bit, i.e., the first bit length is 32 bits, and the format could also be represented as 4 bytes of "00 00 00 01".

[0099] In some embodiments, determining one or more first units in the bitstream based on one or more first indication information in the bitstream includes: determining the starting position of the current first unit based on the current first indication information in the bitstream; and acquiring the current first unit starting from the starting position of the current first unit.

[0100] The current first unit is the first unit to be extracted from one or more first units in the bitstream. Each first unit is quickly located by scanning the start prefix code of the first units in the bitstream, thus correctly decoding the data of the first unit and enabling random access to any first unit in the bitstream. Furthermore, when the decoder or system layer detects packet loss or bit errors, it can discard the erroneous first unit and scan for the next complete and error-free first unit to begin correct decoding.

[0101] In some embodiments, the method further includes one of the following: determining the data packet size of the current first unit based on the current first indication information and the next first indication information; determining the data packet size of the current first unit based on the current second indication information and the third indication information; correspondingly, obtaining the current first unit from the starting position of the current first unit includes: obtaining the current first unit from the starting position of the current first unit based on the data packet size of the current first unit.

[0102] In some embodiments, acquiring the current first unit from the starting position of the current first unit includes: acquiring the current first unit from the starting position of the current first unit; determining that the acquisition of the current first unit is complete when the next first indication information is detected; or, acquiring the current first unit from the starting position of the current first unit; determining that the acquisition of the current first unit is complete when the third indication information is detected.

[0103] The next first and third indication messages are used to indicate the end position of the current first unit.

[0104] For example, the third indication information can be the start prefix code of the third unit. The third unit is different from the first and second units, and the start prefix code of the third unit is different from the start prefix code of the first unit and the start prefix code of the second unit. As another example, the third indication information can also be used to indicate the end position of the bitstream. That is, the first unit is obtained in the bitstream from after the start prefix code corresponding to the current first unit and before the next start prefix code, or the first unit is obtained in the bitstream from after the start prefix code of the first unit and before the third indication information or the third indication information.

[0105] In some embodiments, the method further includes: parsing the header information of the first unit to determine the data type identifier of the first unit; and, if the data type identifier of the first unit is a third preset value, parsing the load data of the first unit to obtain one or more second units corresponding to the preset data type in the first unit.

[0106] The first data type identifier is used to indicate the data type of the first unit, and the third preset value is any one of one or more preset values, which are used to indicate one or more preset data types.

[0107] If the data type of the first unit is a preset data type, it means that the first unit includes data packets or data units encapsulated in the form of the second unit. If the data type of the first unit is not a preset data type, it means that the second unit does not include data packets or data units encapsulated in the form of the second unit.

[0108] In some embodiments, if the data type identifier of the first unit is not a third preset value, the payload data of the second unit is parsed to obtain one or more video parameter sets. That is, the video parameter sets are directly carried in the first unit and do not need to be encapsulated into the second unit, which enables rapid extraction and parsing of the video parameter sets.

[0109] In some embodiments, if the value of the data type identifier of the first unit is not a third preset value, the header information of the first unit is parsed to determine the video parameter set index of the first unit. The video parameter set index is used to indicate the video parameter set referenced when parsing the payload data of the first unit.

[0110] In some embodiments, a third preset value is used to indicate a preset data type, which includes reconstruction information and image information.

[0111] For example, the reconstruction information includes first reconstruction information that does not support random access and second reconstruction information that supports random access, and the image information includes first image information that does not support random access and second image information that supports random access. The second image information that supports random access may be image information including I-frames, and the second reconstruction information that supports random access may be reconstruction information including I-frames.

[0112] For example, the image information is image feature information. For example, the image feature information includes first image feature information that does not support random access and second image feature information that supports random access.

[0113] In some embodiments, the reconstruction information includes one or more data types, and / or, the image information includes one or more data types.

[0114] For example, the first reconstruction information that does not support random access includes one or more data types, the second reconstruction information that supports random access includes one or more data types, the first image information that does not support random access includes one or more data types, and the second image information that supports random access includes one or more data types.

[0115] For example, the first unit takes the VCM unit as an example. The syntax element structure of the VCM unit is as follows:

[0116] Here, numBytesInVCMUnit represents the data packet size of the current VCM unit. For example, the value of numBytesInVCMUnit is determined by the amount of data between the start prefix code and the next start prefix code of the VCM unit.

[0117] The syntax element structure for VCM header and payload is as follows:

[0118] Among them, vuh_unit_type (first data type identifier) ​​indicates the data type of the VCM unit. For example, the data type unit of the VCM unit defines several types such as VPS parameter set (VPS), reconstruction information or reconstruction data (RSD), reconstruction information that supports random access (RSD_RAP), kernel video data (CVD), and kernel video data that supports random access (CVD_RAP).

[0119] `vuh_vps_id` represents the index of a VCM unit of type VPS that is referenced by certain types of VCM units. `vuh_vps_id` specifies the value of `vps_vcm_parameter_set_id` for the effective VCM parameter set. The value of `vuh_vps_id` should be in the range of 0 to 15, but the actual value is between 0 and 4. `vuh_reserved_zero_23bits` and `vuh_reserved_zero_27bits` are reserved bits. `leading_zero_8bits` are the 8 zero bits before the start prefix code, which can be placed before the first VCM unit in the video stream (this syntax element may also not appear in the syntax element structure).

[0120] Here, numBytesInVCMUnitPayload represents the size of the payload data of the current VCM unit. The payload data of the VCM unit is parsed based on the value of numBytesInVCMUnitPayload and the data type of the VCM unit. When vuh_unit_type represents VCM_VPS, the VPS is obtained by parsing the payload data of the VCM unit; when vuh_unit_type represents VCM_RSD or VCM_RAP_RSD, the reconstruction information is obtained by parsing the payload data of the VCM unit; when vuh_unit_type represents VCM_CVD or VCM_RAP_CVD, the kernel video is obtained by parsing the payload data of the VCM unit.

[0121] In some embodiments, the data type identifier of the first unit is used to indicate the data type of the first unit and the data type of the second unit it includes.

[0122] For example, the header information of the first unit is parsed to determine the first data type identifier of the first unit. The first data type identifier is used to indicate the data type of the first unit and the data type of the second unit. The value of the first data type identifier is a third preset value, which determines that the data type of the first unit is a preset data type. The third preset value is any one of one or more preset values, which are used to indicate one or more preset data types.

[0123] For example, the reconstruction information includes Sequence ReStoration Data (SRSD), Pequence ReStoration Data (PRSD), SEI (Search Engine Enhancement) information, and End of Reconstruction Information (EOSS). Correspondingly, the data type identifier (vuh_unit_type) of the VCM unit defines SRDS, PRSD, SEI, EOS, SRDS_RAP, PRSD_RAP, SEI_RAP, and EOS_RAP. For instance, the value of vuh_vps_id should be in the range of 0 to 15, and a richer definition of the first data type identifier is represented by expanding the value of vuh_vps_id. By combining the data type identifier of the first unit with the data type identifier of the second unit therein to form a richer definition of the first data type identifier, more data type information can be quickly parsed from the header of the first unit without incurring additional header information bit consumption.

[0124] The structure of the VCM unit, separated by the start prefix code, is as follows:

[0125] Here, `start_code_prefix_one_3bytes` is a 24-bit start prefix code with a specific value, i.e., the first indication information, and `trailing_zero_8bits` are the 8 zero bits at the end of each VCM unit. `start_code_prefix_one_3bytes` is a mandatory start prefix code, `vcm_unit(numBytesInVCMUnit)` indicates the parsed VCM unit, and `trailing_zero_8bits` are the 8-bit end bits at the end of all VCM units.

[0126] In some embodiments, the first indication information is used to indicate the data packet size of the first unit. For example, parsing the bitstream determines one or more pieces of the first indication information within the bitstream.

[0127] Accordingly, based on one or more first indication information in the bitstream, determining one or more first units in the bitstream includes: determining the data packet size of the current first unit indicated by the current first indication information based on the current first indication information; and obtaining the current first unit in the bitstream based on the data packet size of the current first unit.

[0128] Determining one or more first indication information in a bitstream includes: parsing the current first indication information based on a first value, wherein the first value is used to indicate the number of bits to be decoded in the bitstream; the method further includes: updating the first value based on the number of bits occupied by the current first indication information and the data packet size of the current first unit.

[0129] The first value indicates the number of bits to be decoded in the bitstream, which can be some or all of the bits in the bitstream. The unit of the first value includes, but is not limited to, bytes.

[0130] In some embodiments, parsing the current first indication information based on the first value includes: parsing the bit width indication information of the current first indication information based on the first value; and parsing the current first indication information based on the bit width indication information of the current first indication information. The method further includes: updating the first value based on the number of bits occupied by the bit width indication information of the current first indication information.

[0131] In one embodiment, the method for identifying the data packet size of a VCM unit in a bitstream is as follows:

[0132] Here, vcm_unit_size_minus1 plus 1 (i.e., the first indication information) represents the data packet size of the VCM unit vcm_unit(). vcm_unit_size_minus1 uses a fixed bit width (e.g., 1 byte). numBytes represents the number of bits to be decoded in the bitstream, i.e., the first value. minus vcm_unit_size_minus1+1 represents subtracting the data packet size of vcm_nal_unit(). Subtracting 1 represents subtracting the number of bits occupied by vcm_unit_size_minus1.

[0133] S603: Identify one or more second units in the first unit;

[0134] In some embodiments, determining one or more second units in the first unit includes: determining one or more second indication information in the first unit, the second indication information being used to indicate the second unit; and determining one or more second units in the first unit based on the one or more second indication information in the first unit.

[0135] The second indication information is used to indicate the starting position of the first unit and / or the data packet size.

[0136] In some embodiments, the second indication information is used to indicate the starting position of the second unit in the first unit, so that the decoder can quickly identify the second unit and thus correctly parse the data of the second unit.

[0137] For example, the second indication information is a start prefix code with a second bit length, which differs from the first bit length. For instance, if the first unit has 32 bits, the start prefix code for the first unit could be 31 zero bits followed by a 1 bit, or it could be represented as 4 bytes of "00 00 00 01". Similarly, if the second unit has 24 bits, the start prefix code for the second unit could be 23 zero bits followed by a 1 bit, or it could be represented as 3 bytes of "00 00 01". Another example is a 24-bit first bit and a 32-bit second bit.

[0138] In some embodiments, determining one or more second units in the first unit based on one or more second indication information in the first unit includes: determining the starting position of the current second unit based on the current second indication information in the first unit; and acquiring the current second unit starting from the starting position of the current second unit.

[0139] The current second unit is the second unit to be extracted from one or more second units in the first unit. Each second unit is quickly located by scanning the start prefix code of the second units in the first unit, thus correctly decoding the data of the second unit and enabling random access to any second unit in the first unit. Furthermore, when the decoder or system layer detects packet loss or bit errors, it can discard the erroneous second unit and scan for the next complete and error-free second unit to begin correct decoding.

[0140] In some embodiments, the method further includes one of the following: determining the data packet size of the current second unit based on the current second indication information and the next second indication information; determining the data packet size of the current second unit based on the current second indication information and the next first indication information; determining the data packet size of the current second unit based on the current second indication information and the third indication information;

[0141] Accordingly, obtaining the current second unit from the starting position of the current second unit includes: obtaining the current second unit from the starting position of the current second unit based on the data packet size of the current second unit.

[0142] In some embodiments, acquiring the current second unit starting from the starting position of the current second unit includes: acquiring the current second unit starting from the starting position of the current second unit; determining that the acquisition of the current second unit is complete when the next second indication information is detected; or, acquiring the current second unit starting from the starting position of the current second unit; determining that the acquisition of the current second unit is complete when the next first indication information is detected; or, acquiring the current first unit starting from the starting position of the current first unit; determining that the acquisition of the current second unit is complete when the third indication information is detected.

[0143] The next second instruction, the next first instruction, and the third instruction are used to indicate the end position of the current second unit.

[0144] For example, the third indication information can be the start prefix code of the third unit. The third unit is different from the first and second units, and the start prefix code of the third unit is different from the start prefix code of the first unit and the start prefix code of the second unit. As another example, the third indication information can also be used to indicate the end position of the bitstream. That is, the current second unit is obtained from after the start prefix code of the current second unit and before the start prefix code of the next second unit, or from after the start prefix code of the current second unit and before the start prefix code of the first unit of the next second unit, or from after the start prefix code of the current second unit and before the third indication information.

[0145] In some embodiments, the second indication information is used to indicate the data packet size of the second unit; determining one or more second units in the first unit based on one or more second indication information in the first unit includes: determining the data packet size of the current second unit indicated by the current second indication information based on the current second indication information; and obtaining the current second unit in the bitstream based on the data packet size of the current second unit.

[0146] In some embodiments, determining one or more second indication information in a first unit includes: parsing the current second indication information based on a second value, the second value being used to indicate the number of bits to be decoded in the current first unit; the method further includes: updating the second value based on the number of bits occupied by the current second indication information and the data packet size of the current second unit.

[0147] In some embodiments, parsing the current second indication information based on the second value includes: parsing the bit width indication information of the current second indication information based on the second value; and parsing the current first indication information based on the bit width indication information of the current second indication information. Accordingly, the method further includes: updating the second value based on the number of bits occupied by the bit width indication information of the current second indication information.

[0148] The second value indicates the number of bits to be decoded in the first unit, which can be some or all of the bits in the first unit. The unit of the second value includes, but is not limited to, bytes.

[0149] In some embodiments, the first unit includes two parts: header information and payload data. The payload data of the first unit includes one or more second units. The second value can be the number of bits to be decoded in the payload data of the first unit.

[0150] In some embodiments, the second unit includes two parts: header information and payload data. Exemplarily, the first unit includes, but is not limited to, at least one of the following: a V3C unit, a VCM unit, and an FCM unit. The second unit includes a Network Abstraction Layer (NAL) unit, which is a basic transmission unit in video coding standards such as V3C, VCM, and FCM.

[0151] For example, as shown in FIG8, the first unit is a VCM unit and the second unit is a NAL unit. The VCM unit includes a VCM header and VCM payload data. The VCM payload data includes one or more NAL units. The first indication information is the start prefix code of the first unit, and the second indication information is used to indicate the data packet size of the second unit.

[0152] For example, as shown in FIG9, the first indication information is the start prefix code of the first unit, and the second indication information is the start prefix code of the second unit, and the two start prefix codes have different bit lengths and / or string characteristics.

[0153] For example, the first indication information is used to indicate the data packet size of the first unit, and the second indication information is the start prefix code of the second unit.

[0154] For example, the first indication information is used to indicate the data packet size of the first unit, and the second indication information is used to indicate the data packet size of the second unit.

[0155] For example, taking the first unit as a VCM unit and the second unit as a NAL unit, the data types of the VCM unit include reconstruction information and image information. The reconstruction information VCM unit includes reconstruction information NAL units, and the image information NAL unit VCM unit includes image information NAL units.

[0156] The method for identifying the data packet size of the NAL unit in the reconstruction information sub-stream is as follows:

[0157] Here, rsd_nal_unit_size_minus1 plus 1 (i.e., the second indication information) represents the data packet size of the reconstructed information NAL unit vcm_nal_unit(). rsd_nal_unit_size_minus1 uses a fixed bit width (e.g., 1 byte). numBytes represents the number of bits to be decoded in the reconstructed information VCM unit, i.e., the second value. Subtracting rsd_nal_unit_size_minus1+1 means subtracting the data packet size of vcm_nal_unit(). Subtracting 1 means subtracting the number of bits occupied by rsd_nal_unit_size_minus1.

[0158] The syntax element structure of the reconstructed information NAL unit is as follows:

[0159] The syntax element structure of the NAL unit header information is as follows:

[0160] rbsp_byte[i] is the i-th byte of RBSP. rbsp_byte represents the reconstruction information in the NAL unit. The data type of rbsp_byte is determined by vcm_nal_unit_type.

[0161] The kernel video NAL unit nal_unit() of the kernel video sub-stream maintains the NAL unit syntax element structure of the standard specification that its kernel codec conforms to. For example, when the kernel codec uses the H.266 codec, the kernel video NAL unit nal_unit() should be an H.266 compliant NAL unit.

[0162] The method for identifying the packet size of the kernel video NAL unit in the kernel video sub-stream is as follows:

[0163] Here, `cvd_nal_unit_size_precision_bytes_minus1` plus 1 indicates the number of bits occupied by `cvd_nal_unit_size` (e.g., in bytes). `cvd_nal_unit_size` uses a variable bit width, and `cvd_reserved_zero_5bits` is for byte-aligned data padding. `cvd_nal_unit_size` (i.e., the second indicator information) records the packet size of the kernel video NAL unit `nal_unit()`. The reason why the packet size of the kernel video NAL unit is not represented by a fixed bit width similar to that of the reconstruction information NAL unit is that the data volume of the reconstruction information NAL unit is usually small, and a fixed bit width is sufficient to identify the packet size. However, the data volume of the kernel video NAL unit may vary greatly due to parameter sets or image encoding data, and a fixed bit width is insufficient to identify the packet size.

[0164] numBytes represents the number of bits to be decoded in the kernel video VCM unit, i.e., the second value. Subtracting cvd_nal_unit_size means subtracting the data packet size of nal_unit(). Subtracting cvd_nal_unit_size_precision_bytes_minus1+1 means subtracting the number of bits occupied by cvd_nal_unit_size.

[0165] In some embodiments, the method further includes: taking the value of the first type of second indication information as a first preset value, and determining at least two second units in the first unit based on at least two types of second indication information in the first unit.

[0166] In some embodiments, the method further includes: taking the value of the first type of second indication information as a second preset value, and determining a second unit in the first unit.

[0167] In other words, one or more second indication information includes a first type of second indication information and a second type of second indication information. The first type of second indication information is used to indicate the number of second units, and the second type of second indication information is used to indicate the starting position of the second unit and / or the data packet size.

[0168] For example, the value of the first type of second indication information is 0 (i.e., the first preset value), indicating that the number of second units is 1. The value of the first type of second indication information is 1 (i.e., the second preset value), indicating that the number of second units is greater than 1. For example, the first preset value can also be 1, and the second preset value can also be 0.

[0169] In some embodiments, the header information of the first unit is parsed / scanned to determine the first type of second indication information. That is, the first type of second indication information can be placed in the header information of the first unit.

[0170] In other embodiments, the load data of the first unit is parsed / scanned to determine the first type of second indication information. That is, the first type of second indication information can also be placed in the load data of the first unit.

[0171] In some embodiments, the load data of the first unit is parsed / scanned to determine the second type of second indication information.

[0172] When the first unit contains only one second unit, since there is no need to distinguish the boundaries of multiple second units, the second indication information can be omitted to further save data.

[0173] For example, the syntax element structure of the reconstructed information VCM unit is as follows:

[0174] A value of 1 for `single_nal_unit_flag` (i.e., the first type of second indication information) indicates that the current VCM unit's payload data `restoration_data()` contains only one `vcm_nal_unit` (`numBytes`) of size `numBytes` bytes, and `rsd_nal_unit_size_minus1` (i.e., the second type of second indication information) does not appear. A value of 0 for `single_nal_unit_flag` indicates that the current VCM unit's payload data `restoration_data()` contains more than two `vcm_nal_unit`s (`rsd_nal_unit_size`), the VCM payload data packet size is `numBytes` bytes, and `rsd_nal_unit_size_minus1` appears.

[0175] One possible implementation is that, in the above case, when byte alignment is required, an additional 7 bits need to be padded after single_nal_unit_flag.

[0176] Another possible implementation is to place `single_nal_unit_flag` in the VCM unit header. In this case, `restoration_data()` can directly use the value of `single_nal_unit_flag` from the VCM unit header. The specific syntax element structure is as follows:

[0177] In one embodiment, when the VCM unit contains only one kernel video NAL unit, since it is not necessary to distinguish the boundaries of multiple NAL units, the packet size information can be omitted to further save data. Specifically, the syntax element structure of the kernel video payload is as follows:

[0178] A value of 1 for `single_nal_unit_flag` (i.e., the second type of indication information) indicates that the payload data `coded_video_data()` of the current VCM unit contains only one `nal_unit` (`numBytes`) of size `numBytes`, and `cvd_nal_unit_size_precision_bytes_minus1`, `cvd_reserved_zero_5bits`, and `cvd_nal_unit_size` do not appear. A value of 0 for `single_nal_unit_flag` (i.e., the second type of indication information) indicates that the payload data `coded_video_data()` of the current VCM unit contains only two or more `nal_units` (`cvd_nal_unit_size`), the payload data packet size of the VCM is `numBytes` bytes, and `cvd_nal_unit_size_precision_bytes_minus1`, `cvd_reserved_zero_5bits`, and `cvd_nal_unit_size` appear.

[0179] Similar to the design of reconstruction information, the single_nal_unit_flag can also be placed in the VCM unit header.

[0180] In another embodiment, each VCM unit can be constrained to have exactly one reconstruction information NAL unit or kernel video NAL unit. For example, each VCM unit contains only one reconstruction information NAL unit or kernel video NAL unit. The encoder can encode and encapsulate data in real time, meaning that for each image or image block encoded, a set of VCM units can be encapsulated to carry the reconstruction information NAL unit and kernel video NAL unit without waiting for the encoding of more images or image blocks. In this case, the single_nal_unit_flag can also be omitted.

[0181] The syntactic element structure for reconstructing information can be as follows:

[0182] The syntax element structure of kernel video can be as follows:

[0183] Accordingly, determining one or more second units in the first unit includes: if the first unit includes a second unit, determining one of the second units in the first unit.

[0184] S604: Analyze the second unit and determine the analysis result of the second unit;

[0185] In some embodiments, the parsing result includes reconstruction information and image information. Parsing the second unit to obtain the parsing result of the second unit includes: parsing one or more second units corresponding to the reconstruction information to obtain reconstruction information; and parsing one or more second units corresponding to the image information to obtain a first reconstructed image or first reconstructed image features.

[0186] In some embodiments, the video parameter set referenced by the second unit is determined based on the video parameter set index of the second unit; the second unit is parsed based on the video parameter set referenced by the second unit to determine the parsing result of the second unit.

[0187] For example, one or more second units corresponding to the reconstructed information are parsed based on the video parameter set referenced by the second unit to obtain reconstructed information; a decoder is determined based on the video parameter set referenced by the second unit, and one or more second units corresponding to the image information are parsed based on the selected decoder to obtain a first reconstructed image or first reconstructed image features. For example, the decoder includes VVC, HEVC, and AVC, and one decoder is selected based on the video parameter set.

[0188] In some embodiments, parsing the second unit to obtain the parsing result of the second unit includes: parsing the header information of the second unit to determine the data type of the second unit; and based on the data type of the second unit, parsing the payload data of the second unit to obtain the decoding result corresponding to the data type of the second unit. That is, when the first unit includes two or more second units with different data types, it may also be necessary to parse the data type of the second unit and correctly parse the second unit based on the data type of the second unit.

[0189] In some embodiments, the method further includes: parsing the header information of the first unit to determine a first data type identifier of the first unit; and determining the data type of the first unit and the data type of the second unit based on the first data type identifier of the first unit.

[0190] S605: Based on the analysis results of the second unit, the reconstructed image is obtained.

[0191] Reconstructed images include, but are not limited to, reconstructed videos; reconstructed images can also be single-frame reconstructed images.

[0192] In some embodiments, obtaining a reconstructed image based on the parsing results of the second unit includes: post-processing the first reconstructed image or the features of the first reconstructed image based on the reconstruction information to obtain a post-processed second reconstructed image or the features of the second reconstructed image.

[0193] By employing the above technical solution, for multi-layered data units, the first unit is quickly identified through the first indication information, and the sub-units within the first unit are further quickly identified through the second indication information, thereby correctly decoding the data of the first unit and realizing the function of random access to any first unit and its sub-units in the bitstream. Moreover, when packet loss or bit errors are detected, the erroneous data unit can be discarded, and the next complete and error-free data packet can be scanned to begin correct decoding.

[0194] Furthermore, to illustrate the VCM bitstream structure further, VCM currently organizes the video bitstream using multiple sub-bitstreams. One sub-bitstream is the kernel video sub-bitstream obtained from the kernel encoder in the VCM encoder, and another sub-bitstream is the reconstruction information sub-bitstream generated by the VCM encoder based on the reconstruction information obtained from its preprocessing operations on the video. The kernel video sub-bitstream is decoded by the kernel decoder in the VCM decoder to obtain the decoded video, and the reconstruction information sub-bitstream is processed by the VCM decoder to obtain the reconstructed information. The decoded video, based on the reconstructed information, undergoes reconstruction processing to obtain the reconstructed video. This reconstructed video retains key semantic information and can achieve sufficient task accuracy for machine tasks.

[0195] The decoding method provided by the embodiment of this application when applied to VCM includes: The VCM video bitstream has multiple sub-bitstreams, each sub-bitstream is encapsulated in the form of NAL units, and different sub-bitstreams are split and encapsulated into different types of VCM units. To support random access in real-time encoding or fast recovery decoding ability after transmission errors, the VCM unit uses a start code to identify the segmentation boundary of the unit in the VCM video bitstream, that is, each VCM unit has a start prefix code, which can be 24-bit data the same as the start prefix code used in H.266, etc., or a new type of start prefix code. In this way, the decoder can identify the start position of the VCM unit through the start code in real-time decoding and the next start code to identify the end position of the VCM unit, so that it can find the next complete VCM unit in case of random access or packet loss and error in the bitstream to achieve correct decoding. For the sub-bitstream in the VCM unit, if the sub-bitstream contains multiple NAL units, the boundary of each NAL unit is identified by indicating the packet size. The reason for not using the start code is that on the one hand, the start prefix code of the NAL unit will overlap with the start prefix code of the VCM unit, resulting in the decoder being unable to distinguish whether the unit after the start prefix code is a NAL unit or a VCM unit. On the other hand, since the VCM unit can already enable the decoder to read the VCM unit data from start to end through the identification of the start code, using the packet size to identify the boundary of the NAL unit can occupy less data volume than the start code and does not affect the fast scanning of the NAL unit.

[0196] FIG. 10 is a detailed structural schematic diagram of a VCM bitstream structure in the embodiment of this application. The VCM video bitstream includes:

[0197] Video Parameter Set VCM Unit (VCM Parameter Set Unit), including VCM Video Parameter Set Header Information (VCM_VPS) and VCM Payload (vcm_parameter_set());

[0198] Restoration Data Units VCM Unit, including Restoration Data Units VCM Header Information for Supporting Random Access (VCM_RAP_RSD) and VCM Payload (restoration_data_unit()), Restoration Data Units VCM Header Information without Supporting Random Access (VCM_RSD) and VCM Payload (restoration_data_unit());

[0199] The kernel video VCM (Coded Video Data Units) include kernel video VCM header information (VCM_RAP_CVD) and VCM payload (coded_video_data()) that support random access, and kernel video VCM header information (VCM_CVD) and VCM payload (coded_video_data()) that do not support random access.

[0200] The payload of the reconstruction information VCM unit includes video sequence-level reconstruction information (Sequence Restoration Data) and image-level reconstruction information (Picture Restoration Data). The video sequence-level reconstruction information (Sequence Restoration Data) specifically includes the video sequence-level reconstruction information NAL header (VCM_NAL_SRD) and payload (sequence_restoration_data_rbsp()), while the image-level reconstruction information (Picture Restoration Data) specifically includes the image-level reconstruction information NAL header (VCM_NAL_PRD) and payload (picture_restoration_data_rbsp()).

[0201] The workload of kernel video VCM units that support random access includes NAL units that support random access (VVC IRAP NAL Unit) and NAL units that do not support random access (VVC non-IRAP NAL Unit). The workload of kernel video VCM units that do not support random access includes NAL units that do not support random access (VVC non-IRAP NAL Unit).

[0202] Figure 11 is a schematic diagram of another VCM stream structure in an embodiment of this application. The corresponding VCM stream decoding method is described as follows: A start prefix code is obtained from the stream; VCM units are obtained from the start prefix code to the next start prefix code in the stream; the type of the VCM unit is obtained from the VCM unit; if the VCM unit is a reconstruction information sub-stream unit, reconstruction information NAL units are obtained from the VCM unit. When the reconstruction information sub-stream unit contains multiple NAL units, the data packet size of the NAL units is obtained from the VCM unit to obtain each NAL unit separately, and reconstruction information is obtained from the NAL units; if the VCM unit is a kernel video sub-stream unit, kernel video NAL units are obtained from the VCM unit. When the kernel video sub-stream unit contains multiple NAL units, the data packet size of the NAL units is obtained from the VCM unit to obtain each NAL unit separately, and the decoded kernel video is obtained from the NAL units; based on the reconstruction information, the decoded kernel video is reconstructed to obtain a reconstructed video, which can be used to complete machine tasks to obtain high machine task accuracy.

[0203] The syntax element structure of the VCM unit, the syntax element structure of the VCM unit header and payload, the syntax element structure of the VCM unit separated by the start prefix code, the method for identifying the packet size of the reconstruction information NAL unit in the reconstruction information sub-stream, the syntax element structure of the reconstruction information NAL unit in the reconstruction information sub-stream, the syntax element structure of the reconstruction information NAL unit header, and the method for identifying the packet size of the kernel video NAL unit in the kernel video sub-stream are described in the above embodiments.

[0204] The syntax element structure of the VCM parameter set is as follows:

[0205] The grade level information is as follows:

[0206] The payload data of the VCM NAL unit can be sequence-level reconstruction information, image-level reconstruction information, supplementary enhancement information, termination information, etc., depending on its type information.

[0207] vps_vcm_parameter_set_id indicates the identifier of the video parameter set.

[0208] vps_log2_max_restoration_data_picture_order_cnt_lsb_minus4 indicates the recording rules for the time information of image-level reconstruction data, which are used to deduce the effective time of the reconstruction data according to the rules based on the syntax elements in the image-level reconstruction information.

[0209] The type of kernel video data is identified by ptl_profile_codec_group_idc. For example, a value of 0 for ptl_profile_codec_group_idc indicates AVC, 1 indicates HEVC, and 2 indicates VVC.

[0210] ptl_profile_restoration_idc specifies the combination of tools that need to be used for the bitstream to be decoded. For example, the bitstream may need to be reconstructed using one or more tools such as spatial sampling, region relocation, temporal sampling, and data bit width offset.

[0211] ptl_tier_flag and ptl_level_idc specify the level of decoding capabilities required by the bitstream.

[0212] vuh_unit_type indicates the VCM unit type specified in the table below.

[0213] The decoding method provided in this application embodiment can also be used for FCM. The FCM unit includes the following types: global vision model parameter set (VMPS), feature reconstruction information (RSD), and video feature data (CVD), etc.

[0214] The decoding method applied to FCM includes: obtaining the start prefix code from the bitstream; obtaining FCM units from after the start prefix code to before the next start prefix code in the bitstream; obtaining the type of the FCM unit from the FCM unit; if the FCM unit is a reconstruction information sub-bitstream unit, then obtaining the feature reconstruction information (NAL) unit from the FCM unit; when the reconstruction information sub-bitstream unit contains multiple NAL units, obtaining the packet size of the NAL unit from the FCM unit to obtain each NAL unit separately, and then obtaining the reconstruction information (the parsing result of the NAL unit of the feature reconstruction information type) from the NAL unit. If the FCM unit is a kernel video sub-stream unit, then the kernel video NAL unit is obtained from the FCM unit. When the kernel video sub-stream unit contains multiple NAL units, the data packet size of the NAL unit is obtained from the FCM unit to obtain each NAL unit separately. Then, the video features (the parsing results of the NAL units of the video feature type) are decoded from the NAL units. Based on the feature reconstruction information, the video features are reconstructed to obtain reconstructed features. These reconstructed features contain multiple layers of sub-features at different scales, which can be used in the network after feature extraction in machine intelligence tasks to obtain task analysis results.

[0215] The syntax element structure of the FCM unit is as follows:

[0216] Here, numBytesInFCMUnit represents the data packet size of the current VCM unit. For example, the value of numBytesInVCMUnit is determined by the amount of data between the start prefix code and the next start prefix code of the FCM unit.

[0217] The syntax element structure for FCM header information and payload is as follows:

[0218] Among them, fuh_unit_type (first data type identifier) ​​indicates the data type of the FCM unit. For example, the data type unit of the FCM unit defines several types such as VMPS parameter set, RSD (feature reconstruction information), RSD_RAP (feature reconstruction information that supports random access), CVD (kernel video feature data), and CVD_RAP (kernel video feature data that supports random access).

[0219] The data type indicated by fuh_unit_type can be shown in the table below.

[0220] The FCM unit of type FCM_VMPS contains information for feature reconstruction.

[0221] In one implementation, the decoding method obtains FCM NAL units of various data types from the feature reconstruction data FCM units in the bitstream. Examples of the data types of the FCM NAL units are shown in the table below.

[0222] The `fcm_nal_unit_type` indicates the type of FCM NAL unit: `FCM_NAL_FSPS` represents a sequence-level NAL unit, `FCM_NAL_FPPS` represents an image-level NAL unit, `FCM_NAL_EOSS` represents the end unit of the reconstructed data, `FCM_NAL_SEI` represents auxiliary enhancement information of the reconstructed data, `FCM_NAL_RSV` represents a reserved unit type, and `FCM_NAL_UNSPEC` represents a reserved unit type that can be defined by the user.

[0223] In one implementation, the decoding method obtains a sequence-level parameter set and an image-level parameter set for reconstructing features from the FCM unit of the reconstructed data in the bitstream.

[0224] The decoding method provided in this application, for multi-layered data units, reasonably introduces a start prefix code and data packet size as identifiers for the segmentation boundaries of the first unit (VCM unit or FCM unit) and the second unit (NAL unit) in the bitstream at different levels. This enables the first unit to support the ability to quickly recover decoding capabilities after random access or transmission errors in real-time encoding. That is, the decoder can identify the starting position of the first unit by the start code and the ending position of the first unit by the start prefix code of the next first unit in real-time decoding. In this way, the next complete first unit can be found to achieve correct decoding when random access or packet loss errors occur in the bitstream.

[0225] In another embodiment of this application, see Figure 12, which shows a flowchart of an encoding method provided by an embodiment of this application.

[0226] As shown in Figure 12, the method may include:

[0227] S1201: Perform preprocessing operations on the original image to obtain the information to be encoded;

[0228] S1202: Encode the information to be encoded to obtain one or more second units;

[0229] S1203: Generate a first unit based on the one or more second units;

[0230] S1204: Determine the first instruction information for the first unit;

[0231] S1205: Generate a code stream based on the first indication information of the first unit, wherein the first indication information is used to indicate the first unit.

[0232] In some embodiments, the first indication information is a start prefix code having a first number of bits.

[0233] In some embodiments, the first indication information is used to indicate the data packet size of the first unit.

[0234] In some embodiments, the first indication information of the first unit in the bitstream is adjacent to the first unit and is located before the first unit.

[0235] In some embodiments, generating a first unit based on the one or more second units includes: determining second indication information for the one or more second units, the second indication information being used to indicate the second unit;

[0236] A first unit is generated based on the second indication information of the one or more second units and the one or more second units, wherein the second indication information is used to indicate the second unit.

[0237] In some embodiments, the second indication information of the second unit in the first unit is adjacent to the second unit and is located before the second unit.

[0238] In some embodiments, the second indication information is a starting prefix code having a second number of bits, and the second number of bits is different from the first number of bits.

[0239] In some embodiments, the second indication information is used to indicate the packet size of the second unit.

[0240] In some embodiments, generating the first unit based on the one or more second units includes: when there are at least two second units included in the first unit, determining that the value of the first type of second indication information is a first preset value; and determining at least two second types of indication information; generating the first unit based on the first type of second indication information, the at least two second types of indication information, and the at least two second units.

[0241] In some embodiments, the method may further include: when there is one second unit included in the first unit, determining that the value of the first type of second indication information is a second preset value; generating the first unit based on the first type of second indication information and the one second unit.

[0242] In some embodiments, performing encoding processing on the information to be encoded to obtain one or more second units includes: when the data type of the information to be encoded is a preset data type, performing encoding processing on the information to be encoded to obtain one or more second units; determining the data type of the one or more second units based on the data type of the information to be encoded.

[0243] In some embodiments, generating the first unit based on the second indication information of the one or more second units and the one or more second units includes: generating the payload data of the first unit based on the second indication information of the one or more second units and the one or more second units; determining the data type identifier of the first unit based on the data type of the one or more second units; adding the data type identifier of the first unit to the header information of the first unit.

[0244] In some embodiments, the preset data types include reconstruction information and image information.

[0245] In some embodiments, the reconstruction information includes first reconstruction information that does not support random access and second reconstruction information that supports random access, and the image information includes first image information that does not support random access and second image information that supports random access.

[0246] In some embodiments, the image information is image feature information.

[0247] In some embodiments, the reconstruction information includes one or more data types, and the image information includes one or more data types.

[0248] In some embodiments, the data type identifier of the first unit is used to indicate the data type of the first unit and the data types of the second units included therein.

[0249] In some embodiments, preprocessing operations are performed on the original image to obtain the information to be encoded, including: performing preprocessing operations on the original image to obtain image information and reconstruction information; performing encoding processing on the information to be encoded to obtain one or more second units, including: performing encoding processing on the image information to obtain one or more second units corresponding to the image information; performing encoding processing on the reconstruction information to obtain one or more second units corresponding to the reconstruction information.

[0250] In some embodiments, the method may further include: when the data type of the information to be encoded is a video parameter set, performing encoding processing on one or more video parameter sets to obtain the payload data of the first unit; setting the data type of the first unit to a video parameter set and adding it to the header information of the first unit.

[0251] In some embodiments, the method may further include: when the data type of the information to be encoded is a preset data type, determining the video parameter set referred to by the first unit; determining the video parameter set index of the first unit based on the video parameter set referred to by the first unit; adding the video parameter set index of the first unit to the header information of the first unit.

[0252] Exemplarily, the VCM bitstream encoding method is described as follows:

[0253] For the input video, the encoder performs preprocessing operations to obtain the processed video, and uses the kernel encoder to perform encoding compression on it to obtain the kernel video sub-bitstream, which consists of multiple kernel NAL units;

[0254] The encoder encapsulates the kernel video sub-bitstream into at least one VCM unit, and identifies the type of the VCM unit as the kernel video sub-bitstream type, and records the unit packet size of each kernel NAL unit in the VCM unit to distinguish the boundaries of each kernel NAL unit. This effect can also be achieved by adding a start prefix code;

[0255] The encoder obtains the reconstruction information for the reconstruction post - processing operation of the decoder from the pre - processing operation. This reconstruction information can guide the reconstruction of the decoded video, and encapsulates the reconstruction information into a reconstruction information sub - bitstream. The sub - bitstream contains at least one reconstruction information NAL unit, and then encapsulates the reconstruction information sub - bitstream into at least one VCM unit, identifies the type of the VCM unit as the reconstruction information sub - bitstream type, and records the unit packet size for each reconstruction information NAL unit in the VCM unit to distinguish the boundaries of each reconstruction information NAL unit. This effect can also be achieved by adding a start prefix code;

[0256] The encoder encapsulates relevant information such as the type of the core decoder that the decoder should use into a VCM unit, and the type of this VCM unit is VPS;

[0257] The encoder organizes the VCM units of VPS type, the VCM units of reconstruction information sub - bitstream type, the VCM units of core video sub - bitstream type, and other possible types of VCM units to obtain a VCM video bitstream. The encoder adds a start prefix code to each VCM unit to distinguish the boundaries of the VCM units.

[0258] The above operations do not necessarily need to be carried out in sequence. It is also possible to alternately encode and encapsulate the VCM units of reconstruction information sub - bitstream type, the VCM units of core video sub - bitstream type, etc., to achieve lower - latency encoding.

[0259] Exemplarily, the FCM bitstream encoding method is described as follows:

[0260] For the input video features, the encoder performs pre - processing operations to obtain processed video features, and uses a core encoder to encode and compress them to obtain a video feature sub - bitstream. The video feature sub - bitstream consists of multiple core NAL units;

[0261] The encoder encapsulates the video feature sub - bitstream into at least one FCM unit, identifies the type of the FCM unit as the core video sub - bitstream type, and records the unit packet size for each core NAL unit in the FCM unit to distinguish the boundaries of each core NAL unit. This effect can also be achieved by adding a start prefix code;

[0262] The encoder obtains reconstruction information for the reconstruction post-processing operation of the decoder from the pre-processing operation. This reconstruction information can guide the reconstruction of decoded video features, and the encoder encapsulates the reconstruction information into a reconstruction information sub-stream. This sub-stream contains at least one reconstruction information NAL unit, and then encapsulates the reconstruction information sub-stream into at least one FCM unit. The encoder identifies the type of the FCM unit as the reconstruction information sub-stream type, and records the unit data packet size for each reconstruction information NAL unit in the FCM unit to distinguish the boundaries of each reconstruction information NAL unit. This effect can also be achieved by adding a start prefix code.

[0263] The encoder encapsulates relevant information such as the type of the kernel decoder that the decoder should use into an FCM unit. The type of this FCM unit is VPS.

[0264] The encoder organizes the FCM unit of VPS type, the FCM unit of reconstruction information sub-stream type, the FCM unit of kernel video sub-stream type, and other possible types of FCM units to obtain an FCM video feature stream. The encoder adds a start prefix code to each FCM unit to distinguish the boundaries of the FCM units.

[0265] The above operations do not necessarily need to be carried out in sequence. The encoding and encapsulation of the FCM unit of reconstruction information sub-stream type, the FCM unit of kernel video sub-stream type, etc. can also be carried out alternately to achieve lower-delay encoding.

[0266] In another embodiment of the present application, based on the same inventive concept as the foregoing embodiment, referring to FIG. 13, which shows a schematic structural diagram of an encoder provided by an embodiment of the present application. As shown in FIG. 13, the encoder 1300 may include a first processing unit 1301 and an encoding unit 1302; wherein,

[0267] The first processing unit is configured to perform pre-processing operations on the original image to obtain information to be encoded;

[0268] The encoding unit is configured to perform encoding processing on the information to be encoded to obtain one or more second units;

[0269] The encoding unit is further configured to generate a first unit based on one or more second units. The second indication information is used to indicate the second unit;

[0270] The encoding unit is further configured to determine the first indication information of the first unit; based on the first indication information of the first unit, generate a code stream. The first indication information is used to indicate the first unit.

[0271] It can be understood that each functional unit of the encoder also executes the encoding method of any one of the foregoing embodiments.

[0272] Understandably, in the embodiments of this application, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and can also be a module or a non-modular one. Furthermore, the components in this embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit described above can be implemented in hardware or as a software functional module.

[0273] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0274] Therefore, embodiments of this application provide a computer-readable storage medium applied to an encoder 1300, the computer-readable storage medium storing a computer program that, when executed by a first processor, implements the method of any of the foregoing embodiments.

[0275] This application provides a computer-readable storage medium that stores a bitstream generated by an encoding method such as described above.

[0276] Based on the composition of the encoder 1300 and the computer-readable storage medium, referring to Figure 14, a schematic diagram of the specific hardware structure of the encoder 1300 provided in this embodiment of the application is shown. As shown in Figure 14, the encoder 1300 may include: a first communication interface 1401, a first memory 1402, and a first processor 1403; the various components are coupled together through a first bus system 1404. It is understood that the first bus system 1404 is used to realize the connection and communication between these components. In addition to a data bus, the first bus system 1404 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as the first bus system 1404 in Figure 14.

[0277] The first communication interface 1401 is used for receiving and sending signals during the process of sending and receiving information with other external network elements;

[0278] The first memory 1402 is used to store computer programs that can run on the first processor 1403;

[0279] The first processor 1403 is used to execute the following when running computer programs:

[0280] Preprocessing operations are performed on the original image to obtain the information to be encoded;

[0281] The information to be encoded is encoded to obtain one or more second units;

[0282] A first unit is generated based on one or more second units, and second indication information is used to indicate the second units;

[0283] Determine the first instruction information for the first unit;

[0284] A code stream is generated based on the first indication information of the first unit, whereby the first indication information is used to indicate the first unit.

[0285] It is understood that the first memory 1402 in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The first memory 1402 of the system and method described in this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0286] The first processor 1403 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the first processor 1403 or by instructions in software form. The first processor 1403 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the first memory 1402. The first processor 1403 reads the information in the first memory 1402 and completes the steps of the above method in conjunction with its hardware.

[0287] It is understood that the embodiments described in this application can be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions of this application, or combinations thereof. For software implementation, the technology of this application can be implemented through modules (e.g., procedures, functions, etc.) that perform the functions of this application. Software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0288] Alternatively, as another embodiment, the first processor 1403 is also configured to execute any of the methods in the foregoing embodiments when running a computer program.

[0289] This embodiment provides an encoder in which, for multi-layered data units, the first unit is quickly identified by adding first indication information, and the sub-units within the first unit are quickly identified by adding second indication information. This enables the decoding end to correctly decode the data of the first unit, achieving random access to any first unit and its sub-units in the bitstream. Moreover, when packet loss or bit errors are detected, the decoding end can discard the erroneous data unit and scan for the next complete and error-free data packet to begin correct decoding.

[0290] In another embodiment of this application, based on the same inventive concept as the foregoing embodiments, referring to FIG15, a schematic diagram of the composition structure of a decoder 1500 provided in an embodiment of this application is shown. As shown in FIG15, the decoder 1500 may include: a decoding unit 1501 and a second processing unit 1502; wherein,

[0291] The decoding unit is configured to determine one or more first indication information in the bitstream, the first indication information being used to indicate a first unit; and to determine one or more first units in the bitstream based on one or more first indication information in the bitstream.

[0292] The decoding unit is also configured to determine one or more second units in the first unit;

[0293] The decoding unit is also configured to parse the second unit and determine the parsing result of the second unit;

[0294] The second processing unit is also configured to obtain the reconstructed image based on the analysis results of the second unit.

[0295] Understandably, each functional unit of the decoder also performs the decoding method of any of the aforementioned embodiments.

[0296] Based on the composition of the decoder 1500 and the computer-readable storage medium, Figure 16 illustrates a schematic diagram of the specific hardware structure of the decoder 1500 provided in this embodiment. As shown in Figure 16, the decoder 1500 may include: a second communication interface 1601, a second memory 1602, and a second processor 1603; the various components are coupled together through a second bus system 1604. It is understood that the second bus system 1604 is used to realize the connection and communication between these components. In addition to a data bus, the second bus system 1604 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as the second bus system 1604 in Figure 16.

[0297] in,

[0298] The second communication interface 1601 is used for receiving and sending signals during the process of sending and receiving information with other external network elements;

[0299] The second memory 1602 is used to store computer programs that can run on the second processor 1603;

[0300] The second processor 1603 is used to execute the following when running computer programs:

[0301] Determine one or more first indication information in the bitstream, the first indication information being used to indicate the first unit;

[0302] Based on one or more first indication information in the bitstream, determine one or more first units in the bitstream;

[0303] Identify one or more second units within the first unit;

[0304] Analyze the second unit to determine the results of the analysis;

[0305] Based on the analysis results of the second unit, the reconstructed image is obtained.

[0306] Alternatively, as another embodiment, the second processor 1603 is also configured to execute any of the methods in the foregoing embodiments when running a computer program.

[0307] It is understood that the second memory 1602 has similar hardware functions to the first memory 1402, and the second processor 1603 has similar hardware functions to the first processor 1403; these will not be described in detail here.

[0308] This embodiment provides a decoder in which, for multi-layered data units, the first unit is quickly identified using first indication information, and the sub-units within the first unit are further quickly identified using second indication information, thereby correctly decoding the data of the first unit and enabling random access to any first unit and its sub-units in the bitstream. Furthermore, when packet loss or bit errors are detected, the decoder can discard the erroneous data unit and scan for the next complete and error-free data packet to begin correct decoding.

[0309] In another embodiment of this application, referring to FIG17, a schematic diagram of the composition structure of an encoding / decoding system provided in an embodiment of this application is shown. As shown in FIG17, the encoding / decoding system 1700 may include an encoder 1701 and a decoder 1702.

[0310] In this embodiment, encoder 1701 can be any of the encoders described in the foregoing embodiments, and decoder 1702 can be any of the decoders described in the foregoing embodiments.

[0311] It should be noted that, in this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0312] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0313] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined to obtain new method embodiments without conflict. The features disclosed in the several product embodiments provided in this application can be arbitrarily combined to obtain new product embodiments without conflict. The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined to obtain new method embodiments or device embodiments without conflict.

[0314] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims. Industrial applicability

[0315] This application provides an encoding / decoding method, encoder, decoder, and storage medium. The method includes: determining one or more first indication information in a bitstream, the first indication information indicating a first unit; determining one or more first units in the bitstream based on the one or more first indication information; determining one or more second indication information in the first unit, the second indication information indicating a second unit; determining one or more second units in the first unit based on the one or more second indication information; parsing the second unit to determine the parsing result of the second unit; and obtaining a reconstructed image based on the parsing result of the second unit. Thus, for multi-layered data units, the first unit is quickly identified through the first indication information, and further, the sub-units within the first unit are quickly identified through the second indication information, thereby correctly decoding the data of the first unit and enabling random access to any first unit and its sub-units in the bitstream. Furthermore, when packet loss or bit errors are detected, the erroneous data unit can be discarded, and the next complete and error-free data packet can be scanned to begin correct decoding.

Claims

1. A decoding method, applied to a decoder, wherein, include: Determine one or more first indication information in the bitstream, wherein the first indication information is used to indicate the first unit; Based on one or more first indication information in the bitstream, one or more first units in the bitstream are determined; Determine one or more second units within the first unit; Analyze the second unit to determine the analysis result of the second unit; Based on the analysis results of the second unit, the reconstructed image is obtained.

2. The method according to claim 1, wherein, The first indication information is a start prefix code with a first number of bits.

3. The method according to claim 2, wherein, The step of determining one or more first units in the bitstream based on one or more first indication information in the bitstream includes: Based on the current first indication information in the bitstream, determine the starting position of the current first unit; The current first unit is obtained starting from its initial position.

4. The method according to claim 3, wherein, The method also includes one of the following: Based on the current first indication information and the next first indication information, determine the data packet size of the current first unit; Based on the current first indication information and the third indication information, determine the data packet size of the current first unit; The step of obtaining the current first unit from its starting position includes: Based on the data packet size of the current first unit, the current first unit is obtained starting from its initial position.

5. The method according to claim 3, wherein, The step of obtaining the current first unit from its starting position includes: The acquisition of the current first unit begins from its starting position; if the next first indication information is detected, the acquisition of the current first unit is determined to be complete; or, The acquisition of the current first unit begins from its starting position; if a third indication is detected, the acquisition of the current first unit is determined to be complete.

6. The method according to claim 1, wherein, The first indication information is used to indicate the data packet size of the first unit; The step of determining one or more first units in the bitstream based on one or more first indication information in the bitstream includes: Based on the current first indication information, determine the data packet size of the current first unit indicated by the current first indication information; Based on the data packet size of the current first unit, the current first unit in the bitstream is obtained.

7. The method according to claim 6, wherein, The determination of one or more first indication information in the bitstream includes: Based on the first value, the current first indication information is parsed, where the first value is used to indicate the number of bits to be decoded in the bitstream; The method further includes updating the first value based on the number of bits occupied by the current first indication information and the data packet size of the current first unit.

8. The method according to claim 7, wherein, The step of parsing the current first indication information based on the first value includes: Based on the first value, parse the bit width indication information of the current first indication information; Based on the bit width indication information of the current first indication information, parse the current first indication information; The method further includes updating the first value based on the number of bits occupied by the current first indication information bit width indication information.

9. The method according to any one of claims 1-8, wherein, Determining one or more second units in the first unit includes: Determine one or more second indication information in the first unit, the second indication information being used to indicate the second unit; Based on one or more second indication information in the first unit, one or more second units in the first unit are determined.

10. The method according to claim 9, wherein, The second indication information is a start prefix code with a second number of bits, which is different from the first number of bits.

11. The method according to claim 10, wherein, Determining one or more second units in the first unit based on one or more second indication information in the first unit includes: Based on the current second indication information in the first unit, determine the starting position of the current second unit; The current second unit is obtained starting from the beginning position of the current second unit.

12. The method according to claim 11, wherein, The method also includes one of the following: Based on the current second indication information and the next second indication information, determine the data packet size of the current second unit; The data packet size of the current second unit is determined based on the current second indication information and the next first indication information; Based on the current second indication information and the third indication information, determine the data packet size of the current second unit; The step of obtaining the current second unit from its starting position includes: Based on the data packet size of the current second unit, the current second unit is obtained starting from the beginning position of the current second unit.

13. The method according to claim 11, wherein, The step of obtaining the current second unit from its starting position includes: The acquisition of the current second unit begins from its starting position; if the next second indication information is detected, the acquisition of the current second unit is determined to be complete; or, The acquisition of the current second unit begins from its starting position; if the next first indication information is detected, the acquisition of the current second unit is determined to be complete; or, The acquisition of the current first unit begins from its starting position; if a third indication is detected, the acquisition of the current second unit is determined to be complete.

14. The method according to claim 4, 5, 12 or 13, wherein, The third indication information is used to indicate the end position of the bitstream.

15. The method according to any one of claims 9, wherein, The second indication information is used to indicate the data packet size of the second unit; Determining one or more second units in the first unit based on one or more second indication information in the first unit includes: Based on the current second indication information, determine the data packet size of the current second unit indicated by the current second indication information; Based on the data packet size of the current second unit, the current second unit in the bitstream is obtained.

16. The method according to claim 15, wherein, Determining one or more second indication messages in the first unit includes: Based on the second value, the current second indication information is parsed, whereby the second value indicates the number of bits to be decoded in the current first unit; The method further includes updating the second value based on the number of bits occupied by the current second indication information and the data packet size of the current second unit.

17. The method according to claim 16, wherein, The step of parsing the current second indication information based on the second value includes: Based on the second value, the bit width indication information of the current second indication information is parsed; Based on the bit width indication information of the current second indication information, parse the current first indication information; The method further includes updating the second value based on the number of bits occupied by the current second indication information bit width indication information.

18. The method according to any one of claims 9-17, wherein, Determining one or more second units in the first unit based on one or more second indication information in the first unit includes: The value of the first type of second indication information is a first preset value. Based on at least two types of second indication information in the first unit, at least two second units in the first unit are determined.

19. The method according to claim 18, wherein, The method further includes: The value of the first type of second indication information is a second preset value, which determines a second unit in the first unit.

20. The method according to any one of claims 1-8, wherein, Determining one or more second units in the first unit includes: If the first unit includes a second unit, then one of the second units in the first unit is determined.

21. The method according to any one of claims 1-20, wherein, The method further includes: Parse the header information of the first unit to determine the data type identifier of the first unit; When the data type identifier of the first unit is set to a third preset value, the load data of the first unit is parsed to obtain one or more second units corresponding to the preset data type in the first unit.

22. The method according to claim 21, wherein, The third preset value is used to indicate the preset data type, which includes reconstruction information and image information.

23. The method according to claim 22, wherein, The reconstruction information includes first reconstruction information that does not support random access and second reconstruction information that supports random access, and the image information includes first image information that does not support random access and second image information that supports random access.

24. The method according to claim 22 or 23, wherein, The image information is image feature information.

25. The method according to claim 22 or 23, wherein, The reconstructed information includes one or more data types, and / or the image information includes one or more data types.

26. The method according to claim 25, wherein, The data type identifier of the first unit is used to indicate the data type of the first unit and the data type of the second unit it includes.

27. The method according to any one of claims 22-24, wherein, The process of parsing the second unit to obtain the parsing result of the second unit includes: The reconstruction information is obtained by parsing one or more second units corresponding to the reconstruction information; Analyze one or more second units corresponding to the image information to obtain a first reconstructed image or first reconstructed image features; The reconstructed image obtained based on the analysis results of the second unit includes: Based on the reconstruction information, the first reconstructed image or the features of the first reconstructed image are post-processed to obtain the post-processed second reconstructed image or the features of the second reconstructed image.

28. The method according to claim 21, wherein, The method further includes: If the value of the data type identifier of the first unit is not a third preset value, the load data of the second unit is parsed to obtain one or more video parameter sets.

29. The method according to claim 21, wherein, The method further includes: When the data type identifier of the first unit takes one or more preset values, the header information of the first unit is parsed to determine the video parameter set index of the first unit. The video parameter set index is used to indicate the video parameter set referenced when parsing the payload data of the first unit.

30. An encoding method applied to an encoder, wherein, include: Preprocessing operations are performed on the original image to obtain the information to be encoded; The information to be encoded is encoded to obtain one or more second units; Based on the one or more second units, a first unit is generated; Determine the first indication information of the first unit; A code stream is generated based on the first indication information of the first unit, wherein the first indication information is used to indicate the first unit.

31. The method according to claim 30, wherein, The first indication information is a start prefix code with a first number of bits.

32. The method according to claim 31, wherein, The first indication information is used to indicate the data packet size of the first unit.

33. The method according to any one of claims 30-32, wherein, The first indication information of the first unit in the bitstream is adjacent to the first unit and is located before the first unit.

34. The method according to any one of claims 30-33, wherein, The generation of the first unit based on the one or more second units includes: Determine second indication information for the one or more second units, the second indication information being used to indicate the second unit; A first unit is generated based on the second indication information of the one or more second units and the one or more second units, wherein the second indication information is used to indicate the second unit.

35. The method according to claim 33, wherein, The second indication information is a start prefix code with a second number of bits, which is different from the first number of bits.

36. The method according to claim 33, wherein, The second indication information is used to indicate the data packet size of the second unit.

37. The method according to any one of claims 34-36, wherein, The second indication information of the second unit in the first unit is adjacent to the second unit and is located before the second unit.

38. The method according to any one of claims 34-37, wherein, The generation of the first unit based on the one or more second units includes: When the first unit includes at least two second units, the value of the first type of second indication information is determined to be a first preset value; and the at least two types of second indication information are determined. The first unit is generated based on the first type of second indication information, the at least two types of second indication information, and the at least two second units.

39. The method according to claim 38, wherein, The generation of the first unit based on the one or more second units further includes: When the first unit includes a second unit, the value of the first type of second indication information is determined to be a second preset value; The first unit is generated based on the first type of second indication information and a second unit.

40. The method according to any one of claims 30-39, wherein, The process of encoding the information to be encoded to obtain one or more second units includes: When the data type of the information to be encoded is a preset data type, the information to be encoded is encoded to obtain one or more second units; Based on the data type of the information to be encoded, the data type of the one or more second units is determined.

41. The method according to claim 40, wherein, The generation of the first unit based on the one or more second units includes: Based on the second indication information of the one or more second units and the one or more second units, the load data of the first unit is generated; Based on the data type of the one or more second units, determine the data type identifier of the first unit; Add the data type identifier of the first unit to the header information of the first unit.

42. The method according to claim 41, wherein, The preset data types include reconstruction information and image information.

43. The method according to claim 42, wherein, The reconstruction information includes first reconstruction information that does not support random access and second reconstruction information that supports random access, and the image information includes first image information that does not support random access and second image information that supports random access.

44. The method according to claim 42 or 43, wherein, The image information is image feature information.

45. The method according to claim 42 or 43, wherein, The reconstructed information includes one or more data types, and the image information includes one or more data types.

46. ​​The method according to claim 45, wherein, The data type identifier of the first unit is used to indicate the data type of the first unit and the data type of the second unit it includes.

47. The method according to any one of claims 42-46, wherein, The preprocessing operation on the original image to obtain the information to be encoded includes: Preprocessing operations are performed on the original image to obtain image information and reconstruction information; The process of encoding the information to be encoded to obtain one or more second units includes: The image information is encoded to obtain one or more second units corresponding to the image information; The reconstructed information is encoded to obtain one or more second units corresponding to the reconstructed information.

48. The method of claim 40, wherein, The method further includes: When the data type of the information to be encoded is a video parameter set, one or more video parameter sets are encoded to obtain the payload data of the first unit; Set the data type of the first unit to the video parameter set and add it to the header information of the first unit.

49. The method according to claim 40, wherein, The method further includes: If the data type of the information to be encoded is a preset data type, the video parameter set referenced by the first unit is determined; Based on the video parameter set referenced by the first unit, determine the video parameter set index of the first unit; Add the video parameter set index of the first unit to the header information of the first unit.

50. An encoder, comprising a first processing unit and an encoding unit; wherein: The first processing unit is configured to perform preprocessing operations on the original image to obtain information to be encoded; The encoding unit is configured to encode the information to be encoded to obtain one or more second units; The encoding unit is further configured to generate a first unit based on the one or more second units; The encoding unit is further configured to determine the first indication information of the first unit; A code stream is generated based on the first indication information of the first unit, wherein the first indication information is used to indicate the first unit.

51. An encoder, comprising a first memory and a first processor; wherein: The first memory is used to store computer programs that can run on the first processor; The first processor is configured to perform the method as described in any one of claims 30 to 50 when running the computer program.

52. A decoder, comprising a decoding unit and a second processing unit; wherein: The decoding unit is configured to determine one or more first indication information in the bitstream, the first indication information being used to indicate a first unit; and to determine one or more first units in the bitstream based on the one or more first indication information in the bitstream. The decoding unit is further configured to determine one or more second units in the first unit; The decoding unit is further configured to parse the second unit and determine the parsing result of the second unit; The second processing unit is further configured to obtain a reconstructed image based on the analysis results of the second unit.

53. A decoder, comprising a second memory and a second processor; wherein: The second memory is used to store computer programs that can run on the second processor; The second processor is configured to perform the method as described in any one of claims 1 to 29 when running the computer program.

54. A computer-readable storage medium, wherein, The computer-readable storage medium stores the bitstream generated by the encoding method as described in any one of claims 30 to 50.

55. A computer-readable storage medium, wherein, The computer-readable storage medium stores a computer program that, when executed, implements the method as described in any one of claims 1 to 29, or the method as described in any one of claims 30 to 50.

Images