Video encoding device, video decoding device, video encoding method, and video decoding method

Abstract

This video encoding device includes: a pre-processing means for executing pre-processing for changing, for each picture, the size of the picture with respect to the size of the input picture; an encoding means for encoding the pre-processed picture; a bit stream generation means for generating a bit stream including data on the encoded pictures; and a size information multiplexing means for multiplexing, onto the bit stream, the maximum output image size of pictures to be output by a post-processing means, which restores the size of a decoded picture obtained by decoding an encoded picture to the size of the input picture, in a device having the post-processing means.

Description

Video encoding apparatus, video decoding apparatus, video encoding method, and video decoding method

[0001] The present disclosure relates to a video encoding apparatus, a video decoding apparatus, a video encoding method, and a video decoding method.

[0002] In order to efficiently transmit or record a video, a video encoding apparatus that generates a coded representation obtained by encoding an input video (hereinafter referred to as a bitstream) and a video decoding apparatus that decodes a bitstream to generate a decoded video are used. When the coded representation obtained by encoding the input video and auxiliary information are clearly distinguished, the coded representation obtained by encoding the input video may be referred to as a video bitstream, and a bitstream related to auxiliary information may be referred to as an auxiliary information bitstream.

[0003] As standards for video encoding schemes, there are H.265 / HEVC (High-Efficiency Video Coding) (see Non-Patent Document 1), H.266 / VVC (Versatile Video Coding) (see Non-Patent Document 2), etc., which are standardized by ITU-T SG16 and ISO / IEC / SC29.

[0004] In these video encoding schemes, video data is managed in a hierarchical structure and is encoded and decoded. The hierarchical structure includes, for example, pictures constituting video data, slices (or tiles) obtained by dividing a picture, coding tree units (CTU: Coding Tree Unit) obtained by dividing a slice, and coding units (CU: Coding Unit) obtained by dividing a coding tree unit.

[0005] The input image of a CU to be processed is normally predictively encoded based on a prediction image generated based on a decoded image that was encoded and decoded earlier than the CU to be processed. That is, a prediction error image obtained by subtracting the prediction image from the input image is encoded and decoded. As predictive encoding, there are intra prediction, which uses a decoded image included in a picture with the same display time as the CU to be processed, and inter prediction, which uses a decoded image included in a picture with a different display time from the CU to be processed.

[0006] A CU encoded based on intra-prediction is called an intra-CU. A CU encoded based on inter-prediction is called an inter-CU.

[0007] A picture that can only contain an intraCU is called an I-picture. A picture that can contain not only an intraCU but also an interCU that uses a decoded image contained in one picture with a different display time than the target CU is called a P-picture. A picture that can contain not only an intraCU and an interCU that uses a decoded image contained in one picture with a different display time than the target CU, but also an interCU that uses decoded images contained in two pictures with different display times than the target CU is called a B-picture.

[0008] The coding of the prediction error image is performed based on frequency conversion, quantization, and entropy coding. The decoding of the prediction error image is performed based on entropy decoding, inverse quantization, and inverse frequency conversion.

[0009] Recommendation ITU-T H.265 "High efficiency video coding", Telecommunication Standardization Sector of ITU, April 2015 Recommendation ITU-T H.266 "Versatile video coding", Telecommunication Standardization Sector of ITU, August 2020 S. Ren et al., "Faster R-CNN: Towards real-time object detection with region proposal networks", IEEE Transactions on Pattern Analysis and Machine, Vol. 39, No. 6, June 2017

[0010] The following are possible ways to utilize the video encoding described above.

[0011] For example, when using VCM (Video Coding for Machines), the video encoder applies preprocessing to the input video to improve the performance of the encoding method, and then encodes the input video using the video encoding method described above (hereinafter referred to as internal video encoding). The video encoder also encodes control information for the preprocessing. Preprocessing includes Temporal sampling, which reduces the number of pictures to encode by downsampling the pictures in the input video; Targeting, which reduces the region of interest (ROI) of a picture and grays out the other areas (changes them to gray); Spatial sampling, which reduces the entire picture; and Bit depth truncation, which discards the lower bit information of the pixel values ​​of the picture. In bit depth truncation, for example, the lower bit information of the luminance component is discarded.

[0012] The video decoder decodes the bitstream using the video encoding method described above (hereinafter referred to as internal video decoding). The video decoder then applies post-processing to the decoded picture to obtain the final decoded image. The video decoder also decodes control information for post-processing. Post-processing includes temporal resampling, which interpolates the picture that was downsampled by time sampling in the video encoder; retargeting, which enlarges the gaze area that was reduced by targeting back to its original size; spatial resampling, which enlarges the picture that was reduced by time sampling back to its original size; and bit depth shift, which restores the brightness of the lower bits that were discarded by bit depth truncation.

[0013] Time resampling is the process that corresponds to time sampling and is equivalent to the inverse process of time sampling. Time resampling restores the time resolution of the picture that was downsampled.

[0014] Retargeting is the process corresponding to targeting, and is equivalent to the inverse process of targeting. Spatial resampling is the process corresponding to spatial sampling, and is equivalent to the inverse process of spatial sampling. Bit depth shifting is the process corresponding to bit depth truncation, and is equivalent to the inverse process of bit depth truncation.

[0015] Spatial sampling and targeting processes allow for variations in the output image size for each picture. Therefore, the video decoder cannot determine the memory allocation for the decoding process. This leads to a problem where interoperability between the video encoder and video decoder is hindered in video encoding schemes that utilize either spatial sampling, targeting, or both.

[0016] The present invention aims to ensure interoperability between a video encoder and a video decoder.

[0017] The video encoding apparatus according to this disclosure includes a preprocessing means that performs preprocessing to change the size of each picture with respect to the size of the input picture; an encoding means that encodes the preprocessed pictures; a bitstream generation means that generates a bitstream containing the data of the encoded pictures; and a postprocessing means that restores the size of the decoded picture obtained by decoding the encoded pictures to the size of the input picture, and a size information multiplexing means that multiplexes the maximum output image size of the pictures output by the postprocessing means into a bitstream.

[0018] The video decoding device according to this disclosure includes decoding means for decoding a bitstream and obtaining a decoded picture, post-processing means for restoring the size of the decoded picture to the size of an input picture in a device having pre-processing means for performing pre-processing to change the size of each picture relative to the size of the input picture, and de-multiplexing means for de-multiplexing the bitstream and obtaining the maximum output image size of the picture output by the post-processing means.

[0019] The video encoding method according to this disclosure includes a device that performs preprocessing to change the size of each picture relative to the size of the input picture, encodes the preprocessed pictures, generates a bitstream containing the data of the encoded pictures, and multiplexes the maximum output image size of the pictures output by the post-processing means to the bitstream, in a device that decodes the encoded pictures and restores the size of the decoded pictures obtained to the size of the input pictures.

[0020] The video decoding method based on this disclosure decodes a bitstream to obtain a decoded picture, restores the size of the decoded picture to the size of the input picture in an apparatus having preprocessing means that performs preprocessing to change the size of each picture relative to the size of the input picture, demultiplexes the bitstream, and obtains the maximum output image size of the picture output by the postprocessing means.

[0021] The video encoding program based on this disclosure causes a computer to perform preprocessing to change the size of each picture relative to the size of the input picture, encode the preprocessed pictures, generate a bitstream containing the data of the encoded pictures, and multiplexes the maximum output image size of the pictures output by the post-processing means in a device that decodes the encoded pictures and restores the size of the decoded pictures obtained to the size of the input pictures to the size of the input pictures.

[0022] The video decoding program based on this disclosure causes a computer to decode a bitstream and obtain decoded pictures, restore the size of the decoded pictures to the size of the input pictures in a device having preprocessing means that performs preprocessing to change the size of each picture relative to the size of the input pictures, demultiplex the bitstream, and obtain the maximum output image size of the pictures output by the postprocessing means.

[0023] According to the present invention, interoperability between a video encoder and a video decoder can be ensured.

[0024] This is a block diagram of a video encoder and video decoder that encode and decode each picture constituting video data. This is a block diagram of an example configuration of the pre-processing section in a video encoder. This is a block diagram of an example configuration of the post-processing section in a video decoder. This is an explanatory diagram showing spatial sampling and spatial resampling. This is an explanatory diagram showing targeting processing. This is an explanatory diagram showing targeting processing. This is an explanatory diagram showing retargeting processing. This is a flowchart showing an example of the operation of a video encoder. This is a flowchart showing an example of the operation of a video decoder. This is an explanatory diagram showing an example of a sequence parameter set related to spatial sampling and targeting. This is an explanatory diagram showing an example of a picture parameter set related to spatial sampling and targeting. This is an explanatory diagram showing an example of syntax related to spatial sampling. This is an explanatory diagram showing an example of syntax related to targeting. This is an explanatory diagram showing an example of processing for time sampling. This is an explanatory diagram showing an example of processing for time resampling. This is a block diagram of an example configuration of an information processing system. This is a block diagram of the main parts of a video encoding device. This is a block diagram of the main parts of a video decoding device.

[0025] The embodiments will be described below with reference to the drawings.

[0026] Embodiment 1. Figure 1 is a block diagram showing a video encoder 100 and a video decoder 200 that encode and decode each picture constituting video data.

[0027] The video encoder 100 includes a pre-processing unit 101, an internal video encoder (internal encoder) 102, a multiplexer 103, and a control unit 104.

[0028] The preprocessing unit 101 performs preprocessing. The internal video encoder 102 encodes the video signal supplied from the preprocessing unit 101 and outputs a video bitstream. The control unit 104 generates an auxiliary information bitstream containing control information for post-processing in the video decoder 200. The multiplexer 103 multiplexes the video bitstream and the auxiliary information bitstream and outputs it. Therefore, the control information for post-processing is signaled to the video decoder 200.

[0029] The video decoder 200 includes a post-processing unit 201, an internal video decoder (internal decoder) 202, a demultiplexer 203, and a control unit 204.

[0030] The demultiplexer 203 demultiplexes the bitstream to obtain a video bitstream and an auxiliary information bitstream. The internal video decoder 202 decodes the video bitstream supplied from the demultiplexer 203 to obtain a decoded video using internal video encoding. The control unit 204 decodes the auxiliary information bitstream supplied from the demultiplexer 203 to obtain control information for post-processing. The post-processing unit 201 performs post-processing according to the control information.

[0031] Note that the arrows in Figure 1 simply indicate the direction of signal (data) flow, but do not exclude bidirectional flow. This is also true for other block diagrams.

[0032] [Explanation of Pre-processing and Post-processing] Figure 2 is a block diagram showing an example configuration of the pre-processing unit 101 in the video encoder 100. Figure 3 is a block diagram showing an example configuration of the post-processing unit 201 in the video decoder 200.

[0033] The preprocessing unit 101 includes a time sampling unit 1011, a targeting unit 1012, a spatial sampling unit 1013, and a bit depth truncation unit 1014. The output of the bit depth truncation unit 1014 is supplied to the internal video encoder 102. Auxiliary information is supplied to the multiplexer 103. Note that the auxiliary information may also be supplied to the multiplexer 103 via the control unit 104.

[0034] The time sampling unit 1011 performs the time sampling described above. The targeting unit 1012 performs the targeting process described above. The spatial sampling unit 1013 performs the spatial sampling described above. The bit depth truncation unit 1014 performs the bit depth truncation described above. Note that the execution order of time sampling, targeting process, spatial sampling, and bit depth truncation may differ from the order shown in Figure 2.

[0035] Furthermore, although both the targeting unit 1012 and the spatial sampling unit 1013 are clearly shown in Figure 2, the video encoder 100 utilizes either spatial sampling or targeting processing, or both.

[0036] The post-processing unit 201 includes a time resampling unit 2011, a retargeting unit 2012, a spatial resampling unit 2013, and a bit depth shift unit 2014.

[0037] In the video decoder 200, the time resampling unit 2011 performs the time resampling described above. The retargeting unit 2012 performs the retargeting process described above. The spatial resampling unit 2013 performs the spatial resampling described above. The bit depth shift unit 2014 performs the bit depth shift described above. Note that the execution order of time resampling, retargeting process, spatial resampling, and bit depth shift may differ from the order shown in Figure 3.

[0038] Next, spatial sampling and spatial resampling will be explained in more detail with reference to Figure 4.

[0039] The spatial sampling unit 1013 in the preprocessing unit 101 derives a picture magnification preferred for a machine processing task. As a machine processing task, object detection, as described in Non-Patent Document 3, can be used. As illustrated in Figure 4, the spatial sampling unit 1013 reduces the picture by the derived magnification. As a result, the amount of code in the bitstream is reduced. The information regarding the picture magnification obtained through spatial sampling is multiplexed into the bitstream as auxiliary information.

[0040] In the post-processing unit 201, the spatial resampling unit 2013 enlarges a picture to its original size, as illustrated in FIG. 4, based on information about the scaling factor of the picture demultiplexed from the bitstream. As a result, the spatial resolution of the reduced picture is restored.

[0041] Next, targeting processing and retargeting processing will be described in more detail.

[0042] In the targeting processing, the targeting unit 1012 reduces a gaze region at each scaling factor, and changes the color of other regions (non-gaze regions) to, for example, gray. Note that gray is merely an example. An effect of reducing the code amount of the bitstream can be obtained through targeting. Information regarding the position and scaling factor of each gaze region obtained by the targeting processing is multiplexed into the bitstream as auxiliary information.

[0043] In the retargeting processing, the retargeting unit 2012 restores the gaze region to its original size based on the position and scaling factor information of each gaze region demultiplexed from the bitstream. Through the retargeting processing, the spatial resolution of the reduced gaze region is restored.

[0044] Note that, as can be understood from variables representing image sizes before and after retargeting processing, it is also possible to enlarge the size of the entire screen through retargeting processing.

[0045] With reference to FIGS. 5A to 5C, targeting processing performed by the targeting unit 1012 in the pre-processing unit 101 will be described. A case where the input picture illustrated in FIG. 5A is input to the pre-processing unit 101 is taken as an example.

[0046] The targeting unit 1012 obtains a gaze region for a machine processing task as illustrated in FIG. 5B. As the machine processing task, the targeting unit 1012 may, for example, use the object detection method described in Non-Patent Document 3. Four rectangular gaze regions are illustrated in FIGS. 5B to 5C. Information indicating the position of each gaze region is included in the auxiliary information.

[0047] In the examples shown in FIGS. 5B to 5C, a magnification of 0.5 indicates that each of the horizontal direction and the vertical direction of the region is reduced to 0.5 times the original size. The same applies to magnifications of 0.75 and 1.00. Note that these magnification values are merely exemplary. In addition, a smaller reduction ratio means a lower degree of reduction. For example, a magnification of 0.75 has a smaller reduction ratio than a magnification of 0.5.

[0048] A larger magnification value indicates a more important region. In other words, the preprocessing unit 101 reduces the size of gaze regions by increasing the magnification for gaze regions assumed to be important regions, and decreasing the magnification for gaze regions that are not assumed to be important. The reduction is achieved, for example, by sample thinning (downsampling). Information indicating the magnification is included in auxiliary information.

[0049] In order to reduce the amount of information in regions other than the gaze region (non-gaze regions), the targeting unit 1012 changes the color of the non-gaze regions to gray, for example. In FIG. 5C, the gray portions are indicated by hatching.

[0050] With reference to FIG. 6, retargeting processing performed by a retargeting unit 2012 in a post-processing unit 201 will be described. Four gaze regions are exemplified in FIG. 6.

[0051] The retargeting unit 2012 enlarges the gaze region in the video exemplified in FIG. 5C based on magnification information at the position indicated by the information indicating the position of the gaze region. The enlargement is achieved, for example, by sample interpolation (upsampling). The information indicating the position of the gaze region and the information indicating the magnification are acquired from an auxiliary information bitstream obtained by demultiplexing from a bitstream.

[0052] Through the processing of the retargeting unit 2012 in the post-processing unit 201, the gaze region in the original image is restored as exemplified in FIG. 6.

[0053] [Explanation on the Encoding Side] The maximum width and height of the picture to be output to the retargeting unit 2012 and the spatial resampling unit 2013 in the video decoder 200 are denoted as output_pic_width_max and output_pic_height_max. The flag indicating that spatial resampling will be used for any picture included in the bitstream is denoted as srd_spatial_resampling_enabled_flag. The flag indicating that retargeting will be used for any picture included in the bitstream is denoted as srd_retargeting_enabled_flag.

[0054] Additionally, the flag indicating the use of spatial resampling for the nth picture of the input video is denoted as prd_spatial_resampling_enabled_flag(n). The width and height before spatial sampling are denoted as spatial_resampled_width(n) and spatial_resampled_height(n). The width and height after spatial sampling are denoted as spatial_resample_width(n) and spatial_resample_height(n). The flag indicating the use of retargeting is denoted as prd_spatial_retargeting_enabled_flag(n). The width and height before targeting are denoted as Rtg_output_pic_width(n) and Rtg_output_pic_height(n). The width and height after targeting are denoted as rtg_image_size_width(n) and rtg_image_size_height(n).

[0055] Note that Rtg_output_pic_width(n) is expressed as the difference between rtg_image_size_width(n) and rtg_to_output_difference_width(n) (Rtg_output_pic_width(n) = rtg_image_size_width(n) + rtg_to_output_difference_width(n)). Similarly, Rtg_output_pic_height(n) is expressed as the difference between rtg_image_size_height(n) and rtg_to_output_difference_height(n) (Rtg_output_pic_height(n) = rtg_image_size_height(n) + rtg_to_output_difference_height(n)).

[0056] As described above, the execution order of time sampling, targeting, spatial sampling, and bit depth truncation in the preprocessing unit 101 may differ from the order shown in Figure 2. However, below, we will use the case where spatial sampling is performed first as an example, with respect to targeting and spatial sampling.

[0057] The spatial sampling unit 1013 performs the spatial sampling described above. The targeting unit 1012 performs the targeting process described above. Then, the preprocessed video signal is supplied from the preprocessing unit 101 to the internal video encoder 102.

[0058] Furthermore, the preprocessor 101 supplies the above-mentioned srd_spatial_resampling_enabled_flag, srd_retargeting_enabled_flag, output_pic_width_max, output_pic_height_max, prd_spatial_resampling_enabled_flag(n), spatial_resample_width(n), spatial_resample_height(n), spatial_resampled_width(n), spatial_resampled_height(n), prd_spatial_retargeting_enabled_flag(n), rtg_image_size_width(n), rtg_image_size_height(n), rtg_to_output_difference_width(n), and rtg_to_output_difference_height(n) to the control unit 104 as auxiliary information. The control unit 104 generates an auxiliary information bitstream containing control information for post-processing in the video decoder 200.

[0059] Figure 7 is a flowchart showing an example of the operation of the video encoder 100. The overall operation of the video encoder 100 will be explained with reference to Figure 7.

[0060] In the video encoder 100, the preprocessing unit 101 performs the preprocessing described above, namely time sampling, targeting, spatial sampling, and bit depth truncation (step S101). The internal video encoder 102 encodes the bit depth shifted video data supplied from the preprocessing unit 101 to generate a video bitstream (step S102).

[0061] The control unit 104 generates an auxiliary information bitstream (step S103). The auxiliary information bitstream includes, as auxiliary information, the parameters of each process in the preprocessing and parameters that can identify the processing order of each process constituting the postprocessing. The multiplexer 103 multiplexes the video bitstream and the auxiliary information bitstream (step S104).

[0062] [Decoding side explanation] In the post-processing unit 201, the spatial resampling unit 2013 performs the spatial resampling described above on the decoded video that has been internally encoded and supplied from the internal video decoder 202, based on the control information supplied from the control unit 204. The retargeting unit 2012 also performs the retargeting process described above based on the control information supplied from the control unit 204. The control unit 204 obtains control information from the auxiliary information bitstream. The control information includes information that can specify the maximum width and maximum height (hereinafter also referred to as the maximum output image size) of the picture to be output to the spatial resampling unit 2013 and the retargeting unit 2012, respectively, in the video decoder 200.

[0063] Figure 8 is a flowchart showing an example of the operation of the video decoder 200. The overall operation of the video decoder 200 will be explained with reference to Figure 8.

[0064] In the video decoder 200, the demultiplexer 203 demultiplexes the bitstream, supplies the video bitstream to the internal video decoder 202, and supplies the auxiliary information bitstream to the control unit 204 (step S201). The internal video decoder 202 decodes the video bitstream to obtain the decoded video (step S202).

[0065] The control unit 204 obtains control information related to post-processing from the auxiliary information bitstream (step S203). The control unit 204 assigns the time resampling unit 2011, the retargeting unit 2012, the spatial resampling unit 2013, and the bit depth shift unit 2014 according to parameters that can identify the processing order included in the control information. Each block constituting the post-processing unit 201 performs post-processing using the control information related to post-processing supplied from the control unit 204 (step S204).

[0066] In this embodiment, the video encoder 100 multiplexes the maximum output image size into a bitstream. That is, the video encoder 100 multiplexes information that allows for the identification of the maximum output image size, obtained through spatial resampling and retargeting processing, into a bitstream.

[0067] In the video decoder 200, the memory allocation for processing can be determined based on the maximum output image size. That is, the post-processing unit 201 (particularly the spatial resampling unit 2013 and the retargeting unit 2012) can smoothly perform spatial resampling and retargeting processing by securing memory space based on the maximum output image size.

[0068] Next, we will explain the syntax related to the maximum output image size. Figures 9 to 12 are explanatory diagrams showing examples of syntax included in the bitstream. Figure 9 is an explanatory diagram showing an example of a sequence parameter set related to spatial sampling and targeting. Figure 10 is an explanatory diagram showing an example of a picture parameter set related to spatial sampling and targeting. Figure 11 is an explanatory diagram showing an example of syntax related to spatial sampling. Figure 12 is an explanatory diagram showing an example of syntax related to targeting.

[0069] The above srd_spatial_resampling_enabled_flag, srd_retargeting_enabled_flag, output_pic_width_max, output_pic_height_max, prd_spatial_resampling_enabled_flag(n), spatial_resample_width(n), spatial_resample_height(n), spatial_resampled_width(n), spatial_resampled_height(n), prd_spatial_retargeting_enabled_flag(n), rtg_image_size_width(n), rtg_image_size_height(n), rtg_to_output_difference_width(n), and rtg_to_output_difference_height(n) are specifically sequence_restoration_data_rbsp() and picture_restoration_data_rbsp() shown in Figures 9 to 12. This includes ), picture_spatial_resampling(), and retargeting_parameters().

[0070] As shown in Figure 9, "sequence_restoration_data_rbsp()" includes the following syntax elements. Note that output_pic_xxx_bit_length, output_pic_width_max, and output_pic_height_max are characteristic syntax elements in this embodiment.

[0071] - srd_sequence_restoration_data_id: Provides an identifier for the VCM SRD for reference by other syntax elements. The value of srd_sequence_restoration_data_id shall be in the range of 0 to 15, inclusive. - vcm_num_units_in_tick: The number of time units of a clock operating at frequency time_scale Hz, corresponding to one increment (called a clock tick) of the clock tick counter on the VCM output. num_units_in_tick is greater than 0. A clock tick in seconds is equal to the quotient obtained by dividing num_units_in_tick by time_scale.For example, if the picture rate of a video signal is 25 Hz, time_scale and num_units_in_tick would be equal to 27,000,000 and 1,080,000, respectively, and consequently, a clock tick would be equal to 0.04 seconds. (It is the number of time units of a clock operating at the frequency time_scale Hz that corresponds to one increment (called a clock tick) of a clock tick counter for the VCM output. num_units_in_tick shall be greater than 0. A clock tick, in units of seconds, is equal to the quotient of num_units_in_tick divided by time_scale.) -vcm_time_scale: This is the number of time units elapsed per second in the VCM output. For example, in a time coordinate system that measures time using a 27MHz clock, time_scale is 27,000,000.The value of time_scale must be greater than 0 (It is the number of time units that pass in one second for the VCM output. For example, a time coordinate system that measures time using a 27 MHz clock has a time_scale of 27,000,000. The value of time_scale shall be greater than 0.). -srd_spatial_resampling_enabled_flag: If equal to 1, specifies that the tool of spatial resampling is enabled. -srd_retargeting_enabled_flag: If equal to 1, indicates that the tool of retargeting is enabled. -output_pic_xxx_bit_length: Specifies the number of bits used to signal output_pic_width_max and output_pic_height_max. - output_pic_width_max: Specifies the maximum width of output pictures after spatial resampling and retargeting.- output_pic_height_max: Specifies the maximum height of output pictures after spatial resampling and retargeting. - srd_temporal_restoration_enabled_flag: If equal to 1, the tool of temporal restoration is enabled. - srd_bit_depth_shift_enabled_flag: If equal to 1, the tool of depth shift is enabled.

[0072] As shown in Figure 10, "picture_restoration_data_rbsp()" contains the following syntax elements.

[0073] - prd_sequence_restoration_data_id: Provides an identifier for the sequence restoration data for reference by other syntax elements. - prd_picture_order_cnt_lsb: Specifies the picture order count of the picture restoration data by the modulo MaxRestorationDataPicOrderCntLsb of the current restoration data. The length of the prd_picture_order_cnt_lsb syntax is equal to Log2MaxRestorationDataPicOrderCntLsb bits. The value of prd_picture_order_cnt_lsb is in the range of 0 to MaxRestorationDataPicOrderCntLsb - 1 (It specifies the picture order count modulo MaxRestorationDataPicOrderCntLsb for the current restoration data. The length of the prd_picture_order_cnt_lsb syntax element is equal to Log2MaxRestorationDataPicOrderCntLsb bits. The value of prd_picture_order_cnt_lsb shall be in the range of 0 to MaxRestorationDataPicOrderCntLsb - 1, inclusive.) - prd_spatial_resampling_enabled_flag: If set to 1, spatial resampling is enabled. If set to 0, spatial resampling is disabled.If prd_spatial_resampling_enabled_flag does not exist, prd_spatial_resampling_enabled_flag is set to 0 (equal to 1 specifies that the tool of spatial resampling is enabled. prd_spatial_resampling_enabled_flag equal to 0 specifies that the tool of spatial resampling is disabled. When prd_spatial_resampling_enabled_flag is not present, prd_spatial_resampling_enabled_flag is set equal to 0.). - prd_retargeting_enabled_flag: If equal to 1, it indicates that retargeting is enabled. If equal to 0, it indicates that retargeting is disabled. If prd_retargeting_enabled_flag does not exist, prd_retargeting_enabled_flag is set to 0 (equal to 1 specifies that the tool of retargeting is enabled. prd_retargeting_enabled_flag equal to 0 specifies that the tool of retargeting is disabled. When prd_retargeting_enabled_flag is not present, prd_retargeting_enabled_flag is set equal to 0.). - prd_temporal_restoration_enabled_flag: If equal to 1, it specifies that temporal resampling is enabled. If set to 0, temporal resampling is disabled.If prd_temporal_restoration_enabled_flag does not exist, prd_temporal_restoration_enabled_flag is set to 0 (equal to 1 specifies that the tool of temporal_restoration is enabled. prd_temporal_restoration_enabled_flag equal to 0 specifies that the tool of temporal_restoration is disabled. When prd_temporal_restoration_enabled_flag is not present, prd_temporal_restoration_enabled_flag is set equal to 0.). - prd_bit_depth_shift_enabled_flag: If equal to 1, it specifies that bit depth shifting is enabled. If set to 0, bit depth shifting is disabled. If prd_bit_depth_shift_enabled_flag does not exist, prd_bit_depth_shift_enabled_flag is set to 0 (equal to 1 specifies that the tool of bit depth shift is enabled. prd_bit_depth_shift_enabled_flag equal to 0 specifies that the tool of bit depth shift is disabled.)

[0074] As shown in Figure 11, "picture_spatial_resampling()" includes the following syntax elements.

[0075] -spatial_resample_width: Specifies the width of an input picture subject to spatial resampling. -spatial_resample_height: Specifies the height of an input picture subject to spatial resampling. -spatial_resampled_width: Specifies the resampled width of the input picture. The resampled width, SR_output_pic_width, is set equal to spatial_resampled_width. -spatial_resampled_height: Specifies the resampled height of the input picture. The resampled height, SR_output_pic_height, is set equal to spatial_resampled_height. -spatial_resample_filter_idx: Indicates the spatial resampling filter index used to designate the spatial resampling filter in the decoder.

[0076] As shown in Figure 12, "retargeting_parameters()" contains the following syntax elements.

[0077] -rtg_image_size_len: Specifies the number of bits used to signal rtg_image_size_width and rtg_image_size_height. -rtg_image_size_width: Specifies the width of an input picture subject to retargeting. -rtg_image_size_height: Specifies the height of an input picture subject to retargeting. -rtg_image_size_difference_flag: If equal to 1, it indicates that the size of pictures output from retargeting is signaled, while equal to 0 indicates that the size of pictures output from retargeting is equal to the size of pictures input to retargeting. -rtg_to_output_difference_len: Specifies the number of bits used for signaling rtg_to_output_difference_width and rtg_to_output_difference_height.(It specifies the number of bits used to signal rtg_to_output_difference_width and rtg_to_output_difference_height.) - rtg_to_output_difference_width: Specifies the difference between the width of the output picture of the retargeting of the input picture and rtg_image_size_width. The width of the output picture, Rtg_output_pic_width, is set equal to rtg_image_size_width + rtg_to_output_difference_width. - rtg_to_output_difference_height: Specifies the difference between the height of the retargeting output picture of the input picture and rtg_image_size_height. The output picture height, Rtg_output_pic_height, is set to equal to rtg_image_size_height + rtg_to_output_difference_height.

[0078] The effects of this embodiment, taking the above-mentioned syntactic elements into consideration, will now be explained.

[0079] In this embodiment, when the srd_spatial_resampling_enabled_flag or srd_retargeting_enabled_flag is enabled, the video encoder 100 includes output_pic_width_max and output_pic_height_max, which indicate the maximum width and height of the picture to be output to the spatial resampling unit 2013 and the retargeting unit 2012, in the bitstream's Sequence restoration data RBSP syntax.

[0080] As a result, the video decoder 200 of this embodiment can determine the memory allocation for decoding based on output_pic_width_max and output_pic_height_max included in the bitstream's Sequence restoration data RBSP syntax, without relying on the picture restoration data RBSP syntax for each picture in the bitstream. Therefore, in video encoding schemes that use spatial sampling or targeting processes to change the output image size for each picture, interoperability between the video encoder and the video decoder can be ensured.

[0081] Embodiment 2. In this embodiment, the interoperability between a video encoder and a video decoder that utilize a video encoding scheme that changes the output image size for each picture is made more reliable by changing the definition of the syntax elements used in the first embodiment.

[0082] In this embodiment, spatial_resampled_width(n), spatial_resampled_height(n), rtg_to_output_difference_width(n), and rtg_to_output_difference_height(n) in the first embodiment are defined as follows. Note that the notation in 『』 indicates the conditions that are weighted to the syntax elements in the first embodiment.

[0083] - spatial_resampled_width: Specifies the resampled width of the input picture. The resampled width, SR_output_pic_width, is set equal to spatial_resampled_width. "It is a bitstream requirement of this standard that SR_output_pic_width shall be less than or equal to output_pic_width_max." - spatial_resampled_height: Specifies the resampled height of the input picture. The resampled height, SR_output_pic_height, is set equal to spatial_resampled_height. "It is a bitstream requirement of this standard that SR_output_pic_width be less than or equal to output_pic_width_height." (It specifies the resampled height of the input picture. The resampled height, SR_output_pic_height, is set equal to spatial_resampled_height. "It is a bitstream requirement of this standard that SR_output_pic_width shall be less than or equal to output_pic_width_height.") - rtg_to_output_difference_width: Specifies the difference between the width of the output picture retargeted from the input picture and rtg_image_size_width.The output picture width, Rtg_output_pic_width, is set to equal to rtg_image_size_width + rtg_to_output_difference_width. "It is a bitstream requirement of this standard that Rtg_output_pic_width must be less than or equal to output_pic_width_max." - rtg_to_output_difference_height: Specifies the difference between the height of the output picture of the retargeting input picture and rtg_image_size_height. The output picture height, Rtg_output_pic_height, is set to be equal to rtg_image_size_height + rtg_to_output_difference_height."It is a bitstream requirement of this standard that Rtg_output_pic_height must be less than or equal to output_pic_width_height." (It specifies the difference in height of the output picture of retargeting of the input picture and rtg_image_size_height. The height of the output picture, Rtg_output_pic_height, is set equal to rtg_image_size_height + rtg_to_output_difference_height. "It is a bitstream requirement of this standard that Rtg_output_pic_height shall be less than or equal to output_pic_width_height.")

[0084] In the video encoder 100, the multiplexer 103 multiplexes the auxiliary information bitstream containing the syntax elements described above. The auxiliary information bitstream contains information regarding spatial sampling, specifically the output image size (output image size of the picture from the spatial resampling unit 2013 in the video decoder 200) which is less than or equal to the maximum output image size (maximum width and maximum height). The auxiliary information bitstream also contains information regarding targeting processing, specifically the output image size (output image size of the picture from the retargeting unit 2012 in the video decoder 200) which is less than or equal to the maximum output image size.

[0085] In the video decoder 200, the demultiplexer 203 demultiplexes the auxiliary information bitstream to obtain the output image size of the spatially resampled image. The demultiplexer 203 also demultiplexes the auxiliary information bitstream to obtain the output image size of the resampling process.

[0086] Therefore, the spatial resampling unit 2013 can restrict the size of the output image as a result of spatial resampling to an output image size less than or equal to the maximum output image size. Similarly, the retargeting unit 2012 can restrict the size of the output image as a result of the retargeting process to an output image size less than or equal to the maximum output image size.

[0087] The configuration of the video encoder and video decoder in this embodiment is the same as the configuration of the video encoder 100 and video decoder 200 in the first embodiment.

[0088] In this embodiment, the video decoder 200 acquires control information that restricts the output image size to be less than or equal to the maximum output image size, based on syntax elements (specifically, spatial_resampled_width, spatial_resampled_height, rtg_to_output_difference_width, and rtg_to_output_difference_height) included in the bitstream from the video encoder 100. Since the spatial resampling unit 2013 and the retargeting unit 2012 can then restrict the output image size to be less than or equal to the maximum output image size, they can reliably determine the memory allocation for the decoding process. For example, the post-processing unit 201 only needs to allocate memory space corresponding to the maximum output image size.

[0089] Embodiment 3. In the first embodiment, the video encoder 100 multiplexes information that can identify the maximum output image size through spatial resampling and retargeting processing into a bitstream, and the video decoder 200 obtains information that can identify the maximum output image size from the bitstream.

[0090] In this embodiment, spatial_resampled_width(n), spatial_resampled_height(n), rtg_to_output_difference_width(n), and rtg_to_output_difference_height(n) in the first embodiment are defined as follows. Note that the notation in 『』 indicates a modified or weighted condition for the syntax elements in the first embodiment.

[0091] - spatial_resampled_width: Specifies the resampled width of the input image. The resampled width, SR_output_pic_width, is set equal to "Min[output_pic_width_max, spatial_resampled_width]". - spatial_resampled_height: Specifies the resampled height of the input image. The resampled height, SR_output_pic_height, is set equal to "Min[output_pic_height_max, spatial_resampled_height]". - rtg_to_output_difference_width: Specifies the difference between the width of the output picture retargeted from the input picture and rtg_image_size_width.The output image width, Rtg_output_pic_width, is set to equal to "Min[output_pic_width_max, rtg_image_size_width + rtg_to_output_difference_width]" (It specifies the difference of the width of the output picture of retargeting of the input picture and rtg_image_size_width. The width of the output picture, Rtg_output_pic_width, is set equal to "Min[output_pic_width_max, rtg_image_size_width + rtg_to_output_difference_width]"). - rtg_to_output_difference_height: Specifies the difference between the height of the output picture of the retargeted input picture and rtg_image_size_height. The output picture height, Rtg_output_pic_height, is set to equal to "Min[output_pic_height_max, rtg_image_size_height + rtg_to_output_difference_height]".

[0092] Min[x, y] is a function that returns the smaller of the two values ​​of x and y.

[0093] The configuration of the video encoder and video decoder in this embodiment is the same as the configuration of the video encoder 100 and video decoder 200 in the first embodiment.

[0094] In this embodiment, in the video decoder 200, the spatial resampling unit 2013 and the retargeting unit 2012 acquire control information that restricts the output image size to be less than or equal to the maximum output image size based on syntax elements (specifically, spatial_resampled_width, spatial_resampled_height, rtg_to_output_difference_width, rtg_to_output_difference_height) included in the bitstream from the video encoder 100.

[0095] Based on the conditions 'Min[output_pic_width_max, spatial_resampled_width]' and 'Min[output_pic_height_max, spatial_resampled_height]', the spatial resampling unit 2013 can ensure that the output picture size is less than or equal to the maximum output image size, even if the specified output image size exceeds the maximum output image size.

[0096] Furthermore, by using `Min[output_pic_width_max, rtg_image_size_width + rtg_to_output_difference_width]` and `Min[output_pic_height_max, rtg_image_size_height + rtg_to_output_difference_height]`, the retargeting unit 2012 can ensure that the output picture size is less than or equal to the maximum output image size, even if the specified output image size exceeds the maximum output image size.

[0097] In other words, the post-processing unit 201 (specifically, the spatial resampling unit 2013 and the retargeting unit 2012) can acquire information obtained when the demultiplexer 203 demultiplexes the bitstream, that is, information to prevent the output image size of the picture output by the spatial resampling unit 2013 and the retargeting unit 2012 from exceeding the maximum output image size.

[0098] Since the spatial resampling unit 2013 and the retargeting unit 2012 can explicitly restrict the output image size to be less than or equal to the maximum output image size, the memory allocation for the decoding process can be reliably determined.

[0099] Therefore, in this embodiment, the constraint that the spatial resampling unit 2013 and the retargeting unit 2012 do not output an image larger than the maximum output image size is more reliably observed. For example, even if the video encoder 100 specifies an output image size that exceeds the maximum output image size, the video decoder 200 can be prevented from outputting a picture larger than the maximum output image size.

[0100] In each of the embodiments described above, when the video encoder 100 enables srd_spatial_resampling_enabled_flag or srd_retargeting_enabled_flag, it multiplexes common syntax elements (output_pic_width_max and output_pic_height_max) that can identify the maximum output image size (maximum width and maximum height) of the picture to be output to the spatial resampling unit 2013 and the retargeting unit 2012 into the bitstream. However, the maximum width and maximum height of the picture to be output to the spatial resampling unit 2013 and the maximum width and maximum height of the picture to be output to the retargeting unit 2012 may be treated as independent auxiliary information. However, the embodiments described above require less coding and less processing for decoding auxiliary information than the form in which auxiliary information related to spatial resampling and auxiliary information related to retargeting processing are treated as independent auxiliary information.

[0101] Next, we will explain in more detail the time sampling that can be performed by the preprocessor 101 and the time resampling and bit depth shift that can be performed by the postprocessor 201.

[0102] Figure 13 is an explanatory diagram showing an example of time sampling processing performed by the time sampling unit 1011. Figure 14 is an explanatory diagram showing an example of time resampling processing performed by the time resampling unit 2011. Note that Figures 13 and 14 illustrate the case where max_temporal_resampling_ratio_idx_minus1=0, that is, the case where MaxTPics=1. In other words, the number of pictures thinned out is 1.

[0103] As illustrated in Figure 13, the amount of encoded data is reduced by performing internal encoding after the picture has been downsampled by time sampling. Furthermore, as illustrated in Figure 14, the downsampled picture is restored in the video encoder 100 by time resampling, for example, multi-stage interpolation.

[0104] Next, we will explain in more detail the bit depth truncation that can be performed by the pre-processing unit 101 and the bit depth shift that can be performed by the post-processing unit 201.

[0105] The bit depth truncation unit 1014 in the preprocessing unit 101 discards the lower bit of the luminance component of the video. This process is equivalent to halving the luminance. This process is also equivalent to shifting the luminance component one bit to the right. Bit depth truncation reduces the amount of data in the bitstream.

[0106] The bit depth shift unit 2014 in the post-processing unit 201 shifts the luminance component in the decoded video one bit to the left. This process is equivalent to doubling the luminance. This process restores the brightness of the video.

[0107] Furthermore, information regarding the number of bits of the luminance component to be truncated by the bit depth truncation unit 1014 is multiplexed into the bitstream as auxiliary information. The bit depth shift unit 2014 then shifts the luminance component to the left based on the bit number information demultiplexed from the bitstream. This control allows for manipulation of any desired number of bits.

[0108] While the above embodiment can be implemented using hardware, it can also be realized using a computer program.

[0109] The information processing system shown in Figure 15 comprises a processor 701 such as a CPU (Central Processing Unit), a program memory 702, a storage medium 703 for storing video data, and a storage medium 704 for storing bitstreams. The storage mediums 703 and 704 may be separate storage mediums or may be storage areas consisting of the same storage medium. Magnetic storage media such as hard disks can be used as storage mediums. Furthermore, the information processing system may comprise multiple processors 701.

[0110] In the information processing system, the program memory 702 stores programs (video encoding programs or video decoding programs) for realizing the functions of each block shown in the above embodiment.

[0111] The processor 701 then performs processing according to the program stored in the program memory 702, thereby realizing the functions of the video encoder 100 or video decoder 200 shown in the above embodiment.

[0112] For example, the processor 701 executes processing according to a video encoding program that realizes the functions of each block in the video encoder 100 shown in Figure 1, thereby realizing the functions of the video encoder 100.

[0113] Furthermore, for example, the processor 701 executes processing according to a video decoding program that realizes the functions of each block in the video decoder 200 shown in Figure 1, thereby realizing the functions of the video decoder.

[0114] Furthermore, at least the program memory 702 is a non-transitory computer-readable medium. However, the program may be stored in various types of transient computer-readable medium. The program is supplied to the transient computer-readable medium, for example, via a wired communication channel or a wireless communication channel, i.e., via electrical signals, optical signals or electromagnetic waves.

[0115] Figure 16 is a block diagram showing the main parts of a video encoding device. The video encoding device 10 shown in Figure 16 (implemented by a video encoder 100 in this embodiment) comprises a preprocessing means 11 (implemented by a preprocessing unit 101 in this embodiment) that performs preprocessing to change the size of each picture relative to the size of the input picture, an encoding means 12 (implemented by an internal video encoder 102 in this embodiment) that encodes the preprocessed pictures, a bitstream generation means 13 (implemented by a multiplexer 103 in this embodiment) that generates a bitstream containing the data of the encoded pictures, and a size information multiplexing means 14 (implemented by a multiplexer 103 in this embodiment) that multiplexes the maximum output image size of the pictures output by the post-processing means in a device having a post-processing means that restores the size of the decoded picture obtained by decoding the encoded pictures to the size of the input picture into a bitstream.

[0116] Figure 17 is a block diagram showing the main parts of the video decoding device. The video decoding device 20 shown in Figure 17 (implemented by a video decoder 200 in this embodiment) comprises: decoding means 21 (implemented by an internal video decoder 202 in this embodiment) that decodes a bitstream and obtains a decoded picture; post-processing means 22 (implemented by a post-processing unit 201 in this embodiment) that restores the size of the decoded picture to the size of the input picture in a device having pre-processing means that performs pre-processing to change the size of each picture relative to the size of the input picture; and de-multiplexing means 23 (implemented by a de-multiplexing unit 203 in this embodiment) that de-multiplexes the bitstream and obtains an output image size that is less than or equal to the maximum output image size of the picture output by the post-processing means 22.

[0117] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications can be made to the configuration and operation of the present disclosure that will be understood by those skilled in the art within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0118] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0119] (Note 1) A video encoding device comprising: preprocessing means that performs preprocessing to change the size of each picture with respect to the size of the input picture; encoding means that encodes the preprocessed pictures; bitstream generation means that generates a bitstream containing the data of the encoded pictures; and size information multiplexing means that multiplexes the maximum output image size of the pictures output by the post-processing means into the bitstream in a device having a post-processing means that restores the size of the decoded pictures obtained by decoding the encoded pictures to the size of the input pictures.

[0120] (Note 2) The preprocessing means is the video encoding device described in Note 1, which performs spatial sampling to reduce the size of the input picture as the preprocessing.

[0121] (Note 3) The preprocessing means is the video encoding device according to Note 1 or Note 2, which performs a targeting process to reduce the size of the gaze area in the input picture as the preprocessing.

[0122] (Note 4) A video decoding device comprising: a decoding means for decoding a bitstream and obtaining a decoded picture; a post-processing means for restoring the size of the decoded picture to the size of the input picture in a device having a pre-processing means for performing pre-processing to change the size of each picture relative to the size of the input picture; and a de-multiplexing means for de-multiplexing the bitstream and obtaining the maximum output image size of the picture output by the post-processing means.

[0123] (Note 5) A video decoding device comprising: a decoding means for decoding a bitstream and obtaining a decoded picture; a post-processing means for restoring the size of the decoded picture to the size of the input picture in a device having a pre-processing means for performing pre-processing to change the size of each picture relative to the size of the input picture; and a de-multiplexing means for de-multiplexing the bitstream and obtaining an output image size less than or equal to the maximum output image size of the picture output by the post-processing means.

[0124] (Note 6) A video decoding device comprising: a decoding means for decoding a bitstream and obtaining a decoded picture; a post-processing means for restoring the size of the decoded picture to the size of the input picture in a device having a pre-processing means for performing pre-processing to change the size of each picture relative to the size of the input picture; and a de-multiplexing means for de-multiplexing the bitstream and obtaining information to prevent the output image size of the picture output by the post-processing means from exceeding the maximum output image size.

[0125] (Note 7) The post-processing means is the video decoding apparatus according to any one of Notes 4 to 6, which performs spatial resampling to enlarge the size of the decoded picture as the post-processing.

[0126] (Note 8) The post-processing means is the video decoding apparatus according to any one of Notes 4 to 6, which performs a retargeting process to enlarge the size of the gaze area in the decoded picture as the post-processing.

[0127] (Note 9) A video encoding method comprising a device having a post-processing means that performs pre-processing to change the size of each picture relative to the size of the input picture, encodes the pre-processed pictures, generates a bitstream containing the data of the encoded pictures, and decodes the encoded pictures to restore the size of the decoded pictures to the size of the input pictures, wherein the maximum output image size of the pictures output by the post-processing means is multiplexed into the bitstream.

[0128] (Note 10) A video decoding method comprising: decoding a bitstream to obtain a decoded picture; restoring the size of the decoded picture to the size of the input picture in an apparatus having preprocessing means that performs preprocessing to change the size of each picture relative to the size of the input picture; and demultiplexing the bitstream to obtain the maximum output image size of the picture output by the postprocessing means.

[0129] (Note 11) A video decoding method comprising: decoding a bitstream to obtain a decoded picture; restoring the size of the decoded picture to the size of the input picture in an apparatus having preprocessing means that performs preprocessing to change the size of each picture relative to the size of the input picture; and demultiplexing the bitstream to obtain an output image size that is less than or equal to the maximum output image size of the picture output by the postprocessing means.

[0130] (Note 12) A video decoding method that decodes a bitstream to obtain a decoded picture, restores the size of the decoded picture to the size of the input picture in a device having preprocessing means that performs preprocessing to change the size of each picture relative to the size of the input picture, and demultiplexes the bitstream to obtain information that prevents the output image size of the picture output by the postprocessing means from exceeding the maximum output image size.

[0131] (Note 13) A video encoding program for a device having a post-processing means that causes a computer to perform pre-processing to change the size of each picture relative to the size of an input picture, encode the pre-processed pictures, generate a bitstream containing the data of the encoded pictures, and decode the encoded pictures to restore the size of the decoded pictures to the size of the input pictures. The program is for multiplexing the maximum output image size of the pictures output by the post-processing means into the bitstream.

[0132] (Note 14) A video decoding program that causes a computer to decode a bitstream and obtain a decoded picture, restore the size of the decoded picture to the size of the input picture in a device having preprocessing means that performs preprocessing to change the size of each picture relative to the size of the input picture, and demultiplex the bitstream to obtain the maximum output image size of the picture output by the postprocessing means.

[0133] (Note 15) A recording medium for recording a bitstream generated by a video encoding device, the device comprising: preprocessing means for performing preprocessing to change the size of each picture relative to the size of the input picture; encoding means for encoding the preprocessed pictures; bitstream generation means for generating a bitstream containing the data of the encoded pictures; and size information multiplexing means for multiplexing the maximum output image size of the pictures output by the post-processing means in a device having a post-processing means for restoring the size of the decoded pictures obtained by decoding the encoded pictures to the size of the input pictures.

[0134] Some or all of the configurations described in Appendix 2 and 3, which are dependent on Appendix 1, and Appendix 3, which is dependent on Appendix 2, or some or all of the configurations described in Appendix 7 and 8, which are dependent on Appendix 4 to 6, may also be dependent on Appendix 9 and 13 or Appendix 10 to 12 in the same dependent relationship as Appendix 2 and 3 or Appendix 7 and 8. Furthermore, not limited to Appendix 1, Appendix 4 to 6, Appendix 9 and 13, or Appendix 10 to 12, some or all of the configurations described as appendices may also be dependent on various hardware, software, various recording means for recording software, or systems, provided that they do not deviate from the embodiments described above.

[0135] Although the present invention has been described above with reference to embodiments and examples, the present invention is not limited to the above embodiments and examples. Various modifications to the structure and details of the present invention can be made, as can be understood by those skilled in the art within the scope of the present invention.

[0136] This application claims priority based on Japanese Patent Application No. 2025-031280, filed on 28 February 2025, and incorporates all of its disclosures herein.

[0137] 10 Video encoding device 11 Preprocessing means 12 Encoding means 13 Bitstream generation means 14 Size information multiplexing means 20 Video decoding device 21 Decoding means 22 Postprocessing means 23 Demultiplexing means 100 Video encoder 101 Preprocessing unit 102 Internal video encoder 103 Multiplexer 104 Control unit 200 Video decoder 201 Postprocessing unit 202 Internal video decoder 203 Demultiplexing unit 204 Control unit 701 Processor 702 Program memory 703, 704 Storage medium 1011 Time sampling unit 1012 Targeting unit 1013 Spatial sampling unit 1014 Bit depth truncation unit 2011 Time resampling unit 2012 Retargeting unit 2013 Spatial resampling unit 2014 Bit depth shift unit

Claims

1. A video encoding device comprising: preprocessing means for performing preprocessing to change the size of each picture relative to the size of the input picture; encoding means for encoding the preprocessed pictures; bitstream generation means for generating a bitstream containing the data of the encoded pictures; and size information multiplexing means for multiplexing the maximum output image size of the pictures output by the postprocessing means into the bitstream, in a device having a postprocessing means for restoring the size of the decoded pictures obtained by decoding the encoded pictures to the size of the input pictures.

2. The video encoding apparatus according to claim 1, wherein the preprocessing means performs spatial sampling to reduce the size of the input picture as the preprocessing.

3. The video encoding apparatus according to claim 1 or 2, wherein the preprocessing means performs a targeting process to reduce the size of the gaze area in the input picture as the preprocessing.

4. A video decoding device comprising: a decoding means for decoding a bitstream and obtaining a decoded picture; a post-processing means for restoring the size of the decoded picture to the size of the input picture in a device having a pre-processing means for performing pre-processing that changes the size of each picture relative to the size of the input picture; and a de-multiplexing means for de-multiplexing the bitstream and obtaining the maximum output image size of the picture output by the post-processing means.

5. A video decoding device comprising: a decoding means for decoding a bitstream and obtaining a decoded picture; a post-processing means for restoring the size of the decoded picture to the size of the input picture in a device having a pre-processing means for performing pre-processing to change the size of each picture relative to the size of the input picture; and a de-multiplexing means for de-multiplexing the bitstream and obtaining an output image size less than or equal to the maximum output image size of the picture output by the post-processing means.

6. A video decoding device comprising: a decoding means for decoding a bitstream and obtaining a decoded picture; a post-processing means for restoring the size of the decoded picture to the size of the input picture in a device having a pre-processing means for performing pre-processing to change the size of each picture relative to the size of the input picture; and a de-multiplexing means for de-multiplexing the bitstream and obtaining information to prevent the output image size of the picture output by the post-processing means from exceeding the maximum output image size.

7. The video decoding apparatus according to any one of claims 4 to 6, wherein the post-processing means performs spatial resampling to enlarge the size of the decoded picture as post-processing.

8. The video decoding apparatus according to any one of claims 4 to 6, wherein the post-processing means performs a retargeting process to enlarge the size of the gaze area in the decoded picture as the post-processing.

9. A video encoding method comprising a device having a post-processing means that performs pre-processing to change the size of each picture relative to the size of the input picture, encodes the pre-processed pictures, generates a bitstream containing the data of the encoded pictures, and decodes the encoded pictures to restore the size of the decoded pictures to the size of the input pictures, wherein the maximum output image size of the pictures output by the post-processing means is multiplexed into the bitstream.

10. A video decoding method comprising: decoding a bitstream to obtain a decoded picture; restoring the size of the decoded picture to the size of the input picture in a device having preprocessing means that performs preprocessing to change the size of each picture relative to the size of the input picture; and demultiplexing the bitstream to obtain the maximum output image size of the picture output by the postprocessing means.

11. A video decoding method comprising: decoding a bitstream to obtain a decoded picture; restoring the size of the decoded picture to the size of the input picture in a device having preprocessing means that performs preprocessing to change the size of each picture relative to the size of the input picture; and demultiplexing the bitstream to obtain an output image size that is less than or equal to the maximum output image size of the picture output by the postprocessing means.

12. A video decoding method that decodes a bitstream to obtain a decoded picture, restores the size of the decoded picture to the size of the input picture in a device having preprocessing means that performs preprocessing to change the size of each picture relative to the size of the input picture, and demultiplexes the bitstream to obtain information that prevents the output image size of the picture output by the postprocessing means from exceeding the maximum output image size.

13. A video encoding program for a device having a post-processing means that causes a computer to perform pre-processing to change the size of each picture relative to the size of an input picture, encode the pre-processed pictures, generate a bitstream containing the data of the encoded pictures, and decode the encoded pictures to restore the size of the decoded pictures to the size of the input pictures. The program is for multiplexing the maximum output image size of the pictures output by the post-processing means into the bitstream.

14. A video decoding program that causes a computer to decode a bitstream and obtain a decoded picture, restore the size of the decoded picture to the size of the input picture in a device having preprocessing means that performs preprocessing to change the size of each picture relative to the size of the input picture, and demultiplex the bitstream to obtain the maximum output image size of the picture output by the postprocessing means.

15. A recording medium for recording a bitstream generated by a video encoding device, the device comprising: pre-processing means for performing pre-processing to change the size of each picture relative to the size of the input picture; encoding means for encoding the pre-processed pictures; bitstream generation means for generating a bitstream containing the data of the encoded pictures; and size information multiplexing means for multiplexing the maximum output image size of the pictures output by the post-processing means in a device having a post-processing means for restoring the size of the decoded pictures obtained by decoding the encoded pictures to the size of the input pictures.

Images