Information processing device, information processing method, and information processing system

Abstract

The present technology pertains to an information processing device, an information processing method, and an information processing system enabling suitable correction of deterioration that occurs in an image due to compression coding. This information processing device is provided with a correction processing unit. With respect to decoded data obtained by decoding a bitstream obtained by compression-coding image data that includes a maximum / minimum pixel which is a pixel for which a maximum value or minimum value is set as the pixel value, the correction processing unit corrects the pixel value of a corresponding pixel, which is a pixel estimated to correspond to the maximum / minimum pixel of the image data, so that the pixel value becomes the maximum value or minimum value. The present technology can be applied, for example, to an electronic apparatus equipped with an image sensor.

Description

Information Processing Apparatus, Information Processing Method, and Information Processing System

[0001] The present technology relates to an information processing apparatus, an information processing method, and an information processing system, and more particularly to an information processing apparatus, an information processing method, and an information processing system that can suitably correct degradation caused in an image by compression encoding.

[0002] Patent Document 1 describes an imaging device including a developing unit that corrects the gradation of a high-luminance region of developed image data based on the saturation level and dynamic range of each color signal in un-developed captured data.

[0003] Japanese Patent Application Laid-Open No. 2015-156616

[0004] Generally, between an image sensor and a companion chip that performs post-stage processing such as development processing, and between the companion chip and a DRAM (Dynamic Random Access Memory), they are connected by a transmission I / F (Interface), and an image captured by the image sensor is transmitted via the transmission I / F. In recent years, with the increase in the resolution, high dynamic range, and high-speed imaging of images, the data volume of images has increased, and the transmission bandwidth used for image transmission in the transmission I / F has sometimes been compressed.

[0005] In order to improve the compression of the transmission bandwidth in the transmission I / F, when applying lossy Rossy compression that causes degradation to an image, the degradation caused by compression encoding is visually more noticeable in the maximum-minimum region where the spatial frequency is low. The maximum-minimum region is a region of pixels (white-out region) where the maximum value (e.g., 255) is set as the pixel value for each component of RGB, or a region of pixels (black crush region) where the minimum value (e.g., 0) is set as the pixel value for each component of RGB.

[0006] The present technology has been made in view of such a situation, and is intended to be able to suitably correct the degradation caused in an image by compression encoding.

[0007] The first aspect of this technology is an information processing device that includes a correction processing unit that corrects the pixel values ​​of corresponding pixels, which are estimated to correspond to the maximum or minimum pixels of the image data, to the maximum or minimum value in the decoded data obtained by decoding a bitstream in which image data including maximum or minimum pixels, which are pixels for which a maximum or minimum value is set as a pixel value, is compressed and encoded.

[0008] The first aspect of this technology is an information processing method which includes correcting the pixel values ​​of corresponding pixels, which are pixels estimated to correspond to the maximum or minimum pixels of the image data, to the maximum or minimum value in the decoded data obtained by decoding a bitstream in which image data including maximum or minimum pixels, which are pixels for which a maximum or minimum value has been set as a pixel value, is compressed and encoded.

[0009] The second aspect of this technology is an information processing system comprising: a first information processing device that includes an encoding unit that compresses and encodes image data including maximum and minimum pixels, which are pixels for which a maximum or minimum value is set as a pixel value, to generate a bitstream; a second information processing device that decodes the bitstream to generate decoded data; and a correction processing unit that corrects the pixel values ​​of corresponding pixels, which are pixels estimated to correspond to the maximum and minimum pixels of the image data, to the maximum or minimum value in the decoded data.

[0010] In the first aspect of this technology, in the decoded data obtained by decoding a bitstream in which image data including maximum and minimum pixels, which are pixels for which a maximum or minimum value is set as a pixel value, is compressed and encoded, the pixel value of the corresponding pixel, which is estimated to be a pixel corresponding to the maximum and minimum pixel of the image data, is corrected to the maximum or minimum value.

[0011] In the second aspect of this technology, a first information processing device compresses and encodes image data including maximum and minimum pixels, which are pixels for which a maximum or minimum value is set as the pixel value, to generate a bitstream. A second information processing device then decodes the bitstream to generate decoded data, and in the decoded data, the pixel values ​​of corresponding pixels, which are pixels estimated to correspond to the maximum and minimum pixels of the image data, are corrected to the maximum or minimum value.

[0012] This is a block diagram illustrating an example of the configuration of an imaging system according to one embodiment of this technology. This diagram illustrates an example of quantization of the pixel value of each pixel in image data. This diagram illustrates pixels referenced when adjusting the white balance. This diagram shows an example of the number of occurrences of pixel values ​​near the saturation value in the original data. This diagram illustrates an example of degradation correction performed by the transmission I / F. This is a block diagram illustrating an example of the configuration of a transmission I / F connecting an image sensor and a companion chip. This is a block diagram illustrating a detailed configuration example of the encoder in Figure 6. This is a block diagram illustrating a detailed configuration example of the decoder in Figure 6. This is a flowchart illustrating the first processing performed by an input I / F having the configuration shown in Figure 6. This diagram illustrates the correspondence between the quantization interval of the pixel value and the pixel value of each pixel in the corrected decoded data. This is a flowchart illustrating the second processing performed by an input I / F having the configuration shown in Figure 6. This diagram shows an example of a table showing the relationship between quantization parameters and the maximum degradation amount. This is a flowchart illustrating the third processing performed by an input I / F having the configuration shown in Figure 6. This diagram shows an example of the number of occurrences of pixel values ​​in the original data, the decoded data before correction, and the decoded data after correction. This is a block diagram illustrating a first modified example of the configuration of a transmission I / F connecting an image sensor and a companion chip. This is a block diagram showing a detailed configuration example of the encoder in Figure 15. This is a block diagram showing a detailed configuration example of the decoder in Figure 15. This is a flowchart illustrating the fourth process performed by an input I / F having a configuration like that of Figure 6 or Figure 15. This is a diagram illustrating the details of quantized pixel values ​​and thresholds. This is a block diagram showing a second modified configuration of the transmission I / F connecting the image sensor and the companion chip. This is a block diagram showing a third modified configuration of the transmission I / F connecting the image sensor and the companion chip. This is a block diagram showing a fourth modified configuration of the transmission I / F connecting the image sensor and the companion chip. This is a block diagram showing a fifth modified configuration of the transmission I / F connecting the image sensor and the companion chip. This is a block diagram showing an example of the configuration of computer hardware.

[0013] The following describes the configurations for implementing this technology. The explanation will proceed in the following order: 1. Overview of the imaging system 2. Configuration and operation of the transmission interface 3. Modified examples

[0014] <1. Overview of the shooting system> Figure 1 is a block diagram showing an example of the configuration of a shooting system 1 according to one embodiment of this technology.

[0015] The imaging system 1 shown in Figure 1 is mounted on electronic devices such as smartphones, tablet devices, mobile phones, HMDs (Head Mounted Displays), digital still cameras, digital video cameras, surveillance cameras, and in-vehicle equipment. As shown in Figure 1, the imaging system 1 consists of an image sensor 11, a companion chip 12, and DRAM.

[0016] The image sensor 11 is configured as a CMOS (Complementary Metal Oxide Semiconductor) image sensor, etc. The image sensor 11 has pixels 31 and an output I / F 32.

[0017] Pixel 31 captures images frame by frame by photoelectrically converting the received light into an electrical signal corresponding to the amount of light. Pixel 31 supplies the captured image data (image data) to the output I / F 32.

[0018] The output interface 32 compresses and encodes the image data supplied from the pixels 31 on a frame-by-frame basis according to a predetermined encoding scheme, and transmits the resulting bitstream to the companion chip 12. Here, for example, lossy compression, which involves degradation, is applied to the image data.

[0019] The companion chip 12 is a chip equipped with a processor such as a DSP (Digital Signal Processor), ISP (Image Signal Processor), and AP (Application Processor), logic circuits, memory, etc. The companion chip 12 has an input I / F 51, an image processing unit 52, a stabilization processing unit 53, a codec processing unit 54, input / output I / Fs 55 to 57, and a bus 58.

[0020] The input interface 51 receives the bitstream transmitted from the output interface 32 of the image sensor 11. The input interface 51 decodes the bitstream according to the encoding scheme used during compression encoding and reconstructs the image data. The input interface 51 corrects the degradation that occurred in the reconstructed image data (decoded data) due to compression encoding and supplies the corrected image data to the image processing unit 52. The output interface 32 of the image sensor 11 and the input interface 51 of the companion chip 12 function as a transmission interface (information processing system) connecting the image sensor 11 and the companion chip 12.

[0021] The image processing unit 52, the stabilization processing unit 53, and the codec processing unit 54 read data stored in the DRAM 13 or store data in the DRAM 13 via the bus 58 when performing their respective processes. For example, these processing units can store data being processed or processed in the DRAM 13, or read data before or during processing from the DRAM 13.

[0022] Input / output interfaces 55 to 57 are provided as interfaces for the image processing unit 52, the stabilization processing unit 53, and the codec processing unit 54 to access the DRAM 13 via the bus 58. Specifically, input / output interfaces 55 to 57 compress and encode the data supplied to the DRAM 13 from each of these processing units, and decode the bitstream read from the DRAM 13 by each of these processing units to reconstruct the image data. Similar to input interface 51, input / output interfaces 55 to 57 correct the degradation that occurs in the reconstructed image data (decoded data) due to compression encoding. In this way, input / output interfaces 55 to 57 function as a transmission interface (information processing system) connecting the companion chip 12 and the DRAM 13.

[0023] The image processing unit 52 includes a pre-processing unit 71, a 3DNR (3-Dimensional Noise Reduction) processing unit 72, and a development processing unit 73.

[0024] The pre-processing unit 71 performs white balance adjustment, clamping, and defective pixel correction processing on the image data supplied from the input I / F 51, and then supplies it to the 3DNR processing unit 72.

[0025] The 3DNR processing unit 72 performs 3DNR processing, which combines 2DNR processing (removing noise in two dimensions by filtering each pixel of the current frame image data supplied from the preprocessing unit 71 with its surrounding pixels) and time-direction processing (weighted averaging of the current frame image data with 3DNR-processed image data from past frames (one frame period or earlier) stored in the DRAM 13). The 3DNR-processed image data of the current frame is overwritten and stored in the DRAM 13, and also supplied to the development processing unit 73.

[0026] The development processing unit 73 performs development processing on the image data supplied from the 3DNR processing unit 72. Development processing is the process of converting, for example, a RAW image (such as a Bayer image) into an image in a common format such as an RGB image or a YUV image.

[0027] The stabilization processing unit 53 performs, for example, electronic image stabilization (EIS) on the developed image data.

[0028] The codec processing unit 54 performs encoding processing for recording or communication and file generation on the image data that has undergone the various processing described above. In the codec processing, it is possible to generate files in formats such as MPEG-2 (MPEG: Moving Picture Experts Group) and H.264 for moving images. It is also possible to generate files in formats such as JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), and GIF (Graphics Interchange Format) for still images.

[0029] Figure 2 illustrates an example of quantization of the pixel value of each pixel in image data.

[0030] The quantization of the pixel values ​​of each pixel in image data is performed assuming, for example, that all pixel values ​​appear with uniform probability, as shown in the upper part of Figure 2. For example, quantization is performed by setting an expected value calculated based on the probability of occurrence of the pixel value as the pixel value of each pixel in the image data.

[0031] In other words, as shown by the dashed line at the bottom of Figure 2, when pixel values ​​are quantized at regular quantization intervals, the pixel value of each pixel in the image data is quantized to the midpoint of the quantization interval (= expected value), as shown by the black dots at the bottom of Figure 2. Therefore, for example, if the pixel values ​​within the interval from A to B are quantized to the same value, then for a pixel that had a value between A and B set as its pixel value in the image data before compression encoding (hereinafter also referred to as the original data), the pixel value in the decoded data will be set to (A + B) / 2. Thus, a quantization error (degradation) can occur between the pixel value of a pixel in the original data and the pixel value of the corresponding pixel in the decoded data in the original data.

[0032] Generally, due to the characteristics of human vision, degradation is less perceptible in areas with high spatial frequencies within an image, and more perceptible in areas with low spatial frequencies. Therefore, degradation caused by compression coding (especially quantization) is more visually noticeable in the maximum and minimum spatial frequency regions. The maximum and minimum regions are areas where the pixel value for each RGB component is set to the maximum value (e.g., 255) (saturated region, blown-out highlight region), or areas where the pixel value for each RGB component is set to the minimum value (e.g., 0) (crushed black region). In the following, pixels in the saturated region will also be referred to as saturated pixels, and pixels in the crushed black region will also be referred to as crushed black pixels.

[0033] Setting pixel values ​​below the saturation value (e.g., (R,G,B) = (255,254,255)) for pixels in the decoded data that correspond to saturated pixels in the original data leads to a decrease in the PSNR (Peak Signal-to-Noise Ratio) of the decoded data. Furthermore, if some pixels in the decoded data that correspond to saturated pixels in the original data have pixel values ​​below the saturation value, saturated and non-saturated pixels will be mixed within the region that was originally the saturated region, making the non-saturated pixels more visually noticeable.

[0034] Furthermore, setting a pixel value less than the saturation value for the pixels in the decoded data corresponding to the saturated pixels in the original data affects, for example, the white balance adjustment performed by the image processing unit 52.

[0035] Figure 3 illustrates the pixels referenced when adjusting the white balance.

[0036] As shown by the dashed ellipse in the upper part of Figure 3, the image processing unit 52 adjusts the white balance of the decoded data by referring to pixels outside the saturation region.

[0037] As shown in the lower part of Figure 3, if a value less than the saturation value is set as the pixel value of a pixel in the decoded data that corresponds to a saturated pixel in the original data, the image processing unit 52 cannot detect the area that was originally a saturated region as a saturated region. Therefore, the image processing unit 52 adjusts the white balance of the decoded data by referring to the pixels of the entire decoded data, including the pixels that were originally saturated pixels. This results in differences in color and saturation levels in the decoded data.

[0038] Figure 4 shows an example of the number of occurrences of pixel values ​​near the saturation value in the original data.

[0039] In the raw data, there is often a statistical bias in the probability of pixel values ​​appearing. For example, in the raw data, as shown in Figure 4, the number of occurrences of the saturation value is overwhelmingly higher among pixel values ​​near the saturation value.

[0040] Figure 5 illustrates an example of degradation correction performed by the transmission interface.

[0041] On the left side of FIG. 5, an example of the number of occurrences of pixel values in the original data is shown in a graph, and on the right side of FIG. 5, the graph for pixel values near the saturation value is shown enlarged.

[0042] As described above, in the original data, the number of occurrences of the saturation value is overwhelmingly large among the pixel values near the saturation value. Therefore, the transmission I / F (input I / F 51 and input / output I / Fs 55 to 57) of the present technology corrects the pixel value of the corresponding pixel, which is a pixel of the decoded data estimated to correspond to the saturated pixel of the original data, to the saturation value. Specifically, as shown onthe right side of FIG. 5, the decoded data is corrected so that the saturation value is set as the pixel value in the decoded data for the pixel of the original data in which a value near the saturation value is set as the pixel value.

[0043] The transmission I / F corrects the pixel value of the corresponding pixel, which is a pixel of the decoded data estimated to correspond to the blacked-out pixel of the original data, to the minimum value.

[0044] In summary, the transmission I / F corrects the pixel value of the corresponding pixel, which is a pixel estimated to correspond to the maximum / minimum pixel of the original data, to the saturation value or the minimum value in the decoded data obtained by decoding the bit stream in which the original data including the maximum / minimum pixel, which is a pixel with the saturation value or the minimum value set as the pixel value, is compression-encoded. Thereby, the saturated area and the blacked-out area in the image data are preserved before and after compression encoding. Therefore, it is expected that the coloring in the saturated area is eliminated and the PSNR of the decoded data is improved. In addition, the image processing unit 52 can detect the area that was the saturated area in the original data, and can appropriately calculate the parameters used for white balance adjustment.

[0045] <2. Configuration and Operation of Transmission I / F> FIG. 6 is a block diagram showing a configuration example of the transmission I / F connecting between the image sensor 11 and the companion chip 12.

[0046] As shown in FIG. 6, the output I / F 32 of the image sensor 11 has an encoder 101.

[0047] The encoder 101 performs intra encoding by compressing the image data of the current frame (original data) using predicted image data generated by referencing the pixel values ​​of pixels in the image data of the current frame, thereby generating a bitstream. The output interface 32 transmits the bitstream to the input interface 51.

[0048] The bitstream includes encoded data, prediction mode information, and quantization parameters. Encoded data is data obtained by compressing and encoding image data. Prediction mode information indicates the prediction mode applied when generating the predicted image data, and quantization parameters are parameters used to quantize the prediction residual (or original data), which is the difference between the image shown in the original data and the predicted image shown in the predicted image data.

[0049] The input interface 51 includes a decoder 111 and a correction processing unit 112.

[0050] The input interface 51 receives the bitstream transmitted from the output interface 32. The decoder 111 performs intra-decoding, decoding the bitstream of the current frame using predicted image data generated by referencing the pixel values ​​of pixels in the image data of the current frame, and then reconstructs (generates) the decoded data. The decoder 111 generates the predicted image data using the predicted mode indicated by the predicted mode information contained in the bitstream. The decoder 111 also performs inverse quantization of the quantized data using the quantization parameters contained in the bitstream.

[0051] The decoder 111 supplies the decoded data, quantization parameters, and prediction mode information to the correction processing unit 112.

[0052] The correction processing unit 112 corrects the degradation that occurred in the decoded data due to compression coding, based on, for example, the quantization parameters and prediction mode supplied from the decoder 111, and outputs the corrected image data.

[0053] Furthermore, input / output interfaces 55 to 57 also have an encoder 101, a decoder 111, and a correction processing unit 112, as described with reference to Figure 6.

[0054] Figure 7 is a block diagram showing a detailed configuration example of the encoder 101.

[0055] The encoder 101 in Figure 7 applies compression encoding to each pixel block obtained by dividing the input image data (original data) of the current frame. Alternatively, compression encoding may be applied to each pixel of the original data.

[0056] As shown in Figure 7, the encoder 101 is composed of an encoding control unit 131, an intra predictor 132, a subtractor 133, a quantization unit 134, an encoding unit 135, an inverse quantization unit 136, an adder 137, a reference buffer 138, and a stream generation unit 139.

[0057] The coding control unit 131 determines the prediction mode to be applied when generating predicted image data and the quantization parameters used for quantizing the prediction residuals, based on the pixel blocks to be compressed and the reference image data supplied from the reference buffer 138. The coding control unit 131 supplies the quantization parameters to the quantization unit 134 and the stream generation unit 139, and supplies the prediction mode information to the intra predictor 132 and the stream generation unit 139.

[0058] The intra predictor 132 performs intra prediction, generating predicted image data from reference image data supplied from the reference buffer 138 in a prediction mode determined by the coding control unit 131. Here, the reference image data is data composed of compressed pixels (pixels that have undergone conversion to prediction residuals and quantization once) surrounding the pixel blocks that are subject to compression coding, from among the pixels of the entire original data. The intra predictor 132 supplies the predicted image data to the subtractor 133 and the adder 137.

[0059] The subtractor 133 subtracts the predicted image data from the pixel block to be compressed and coded to generate a predicted residual, which is then supplied to the quantization unit 134.

[0060] The quantization unit 134 uses the quantization parameters supplied from the coding control unit 131 to quantize the predicted residual supplied from the subtractor 133, and supplies the resulting quantized data to the coding unit 135 and the inverse quantization unit 136.

[0061] The encoding unit 135 entropy encodes the quantized data supplied from the quantization unit 134 and supplies the resulting encoded data to the stream generation unit 139.

[0062] The inverse quantization unit 136 inversely quantizes the quantized data supplied from the quantization unit 134 and supplies the resulting predicted residual to the adder 137.

[0063] The adder 137 reconstructs the pixel block to be compressed and encoded by adding the predicted image data supplied from the intra predictor 132 and the predicted residual supplied from the inverse quantization unit 136. The adder 137 supplies the data of the reconstructed pixel block to the reference buffer 138.

[0064] The reference buffer 138 holds the data supplied from the adder 137 and supplies the reference image data to the encoding control unit 131 and the intra predictor 132 in response to requests from them.

[0065] The stream generation unit 139 combines the quantization parameters and prediction mode information supplied from the coding control unit 131 with the coded data supplied from the coding unit 135 to generate a bitstream.

[0066] Figure 8 is a block diagram showing a detailed configuration example of the decoder 111.

[0067] As shown in Figure 8, the decoder 111 is composed of an extension unit 151, a decoding unit 152, an inverse quantization unit 153, an intra predictor 154, an adder 155, and a reference buffer 156.

[0068] The decompression unit 151 decompresses the bitstream to obtain quantization parameters, prediction mode information, and encoded data. The decompression unit 151 supplies the quantization parameters to the inverse quantization unit 153 and the correction processing unit 112. The decompression unit 151 supplies the prediction mode information to the intra predictor 154 and the correction processing unit 112. The decompression unit 151 supplies the encoded data to the decoding unit 152.

[0069] The decoding unit 152 entropy-decodes the encoded data supplied from the decompression unit 151 and supplies the resulting quantized data to the inverse quantization unit 153.

[0070] The inverse quantization unit 153 uses the quantization parameters supplied from the decompression unit 151 to inverse quantize the quantized data supplied from the decoding unit 152, and supplies the resulting predicted residual to the adder 155.

[0071] The intra predictor 154 performs intra prediction, generating predicted image data from reference image data supplied from the reference buffer 156 in a prediction mode indicated by the prediction mode information supplied from the extension unit 151. Here, the reference image data is data composed of decoded pixels surrounding the pixel block to be decoded, from among the pixels of the entire decoded data. The intra predictor 154 supplies the predicted image data to the adder 155.

[0072] The adder 155 adds the predicted residual supplied from the inverse quantization unit 153 and the predicted image data supplied from the intra predictor 154 to reconstruct the pixel block to be decoded. The adder 155 supplies the data of the reconstructed pixel block as decoded data to the reference buffer 156 and the correction processing unit 112.

[0073] The reference buffer 156 holds the decoded data supplied from the adder 155 and supplies the reference image data to the intra predictor 154 upon request from the intra predictor 154.

[0074] Next, with reference to the flowchart in Figure 9, the first processing performed by the input I / F 51 having the configuration shown in Figure 6 will be described. In Figure 9, the processing performed by the input I / F 51 when the correction processing unit 112 corrects the decoded data based only on the decoded data will be described. For the sake of simplicity, the following description will focus on the processing in which the input I / F 51 corrects the pixel values ​​of the corresponding pixels in the decoded data to the saturation value.

[0075] In step S1, the input interface 51 receives the bitstream transmitted from the output interface 32.

[0076] In step S2, the decoder 111 decodes the bitstream and reconstructs the decoded data.

[0077] In step S3, the correction processing unit 112 defines a predetermined threshold b. Here, the threshold b is prepared in advance for each use case of the original data and the corrected image data, for example, based on statistics of various image data. After the threshold b is defined, the correction processing unit 112 sets the pixels to be corrected from all the pixels of the decoded data.

[0078] In step S4, the correction processing unit 112 obtains the pixel value a of the pixel to be corrected.

[0079] In step S5, the correction processing unit 112 determines whether the pixel value a of the pixel to be corrected is greater than or equal to the threshold b.

[0080] If it is determined in step S5 that the pixel value a is greater than or equal to the threshold b, the correction processing unit 112 identifies the pixel to be corrected as the corresponding pixel of the saturated pixel. Then, in step S6, the correction processing unit 112 performs a correction, setting the saturation value as the corrected pixel value (corrected value) of the pixel to be corrected.

[0081] On the other hand, if it is determined in step S5 that the pixel value a is less than the threshold b, the correction processing unit 112 identifies the pixel to be corrected as a pixel that is not the corresponding pixel of the saturated pixel. Then, in step S7, the correction processing unit 112 performs a correction, using the pixel value a as is to be corrected as the corrected pixel value of the pixel to be corrected.

[0082] It is assumed that errors (degradation) will occur between the pixel values ​​of non-saturated pixels in the original data and the pixel values ​​of the corresponding pixels in the decoded data due to compression coding, including quantization. Therefore, the correction processing unit 112 may apply a correction to the pixel value a to correspond to the degradation caused by compression coding. For example, the correction processing unit 112 may add or subtract the median value of the degradation amount estimated to occur due to compression coding to the pixel value a. Alternatively, for example, the correction processing unit 112 may add or subtract a randomly selected degradation amount within the range from the maximum to the minimum value of the degradation amount estimated to occur due to compression coding to the pixel value a.

[0083] After the processing in step S6 or step S7 is performed, in step S8, the correction processing unit 112 determines whether or not correction processing has been performed on all pixels of the decoded data.

[0084] If it is determined in step S8 that some pixels have not been corrected, the correction processing unit 112 sets the next pixel to be corrected. After the pixel to be corrected is set, the process returns to step S4, and the subsequent processing is repeated.

[0085] On the other hand, if it is determined in step S8 that correction processing has been performed on all pixels, the process ends.

[0086] Through the above process, saturated regions within the image data are preserved before and after compression encoding. This is expected to eliminate color fringing in saturated regions and enable proper white balance adjustment. In other words, the transmission interface can compensate for the degradation of the image caused by compression encoding in accordance with the needs of subsequent processing.

[0087] Figure 10 illustrates the correspondence between the quantization interval of pixel values ​​and the pixel values ​​of each pixel in the corrected decoded data.

[0088] When pixel values ​​are quantized at fixed quantization intervals, as shown enclosed in rectangle #1, for pixels where the original data had a value within the quantization interval including the saturation value, the corrected decoded data will have the saturation value set as the pixel value.

[0089] Furthermore, as shown enclosed in rectangle #2, in the original data, for pixels where the pixel value was set to a value within a quantization interval where the frequency of occurrence of the pixel value is approximated to be approximately uniform, in the corrected decoded data, the pixel value is set to the midpoint of the quantization interval (the expected value of the pixel value within the quantum interval).

[0090] Furthermore, as shown enclosed in rectangle #3, in the original data, for pixels where the pixel value was set to a value within a quantization interval where the number of occurrences of the pixel value changes linearly, in the corrected decoded data, a value different from the midpoint of the quantization interval (the expected value of the pixel value within the quantum interval) is set as the pixel value.

[0091] The expected value of the pixel value within each quantum interval is calculated in advance based on statistics from various image data.

[0092] Next, referring to the flowchart in Figure 11, we will explain the second process performed by the input I / F 51 having the configuration shown in Figure 6. In Figure 11, we will explain the process performed by the input I / F 51 when the correction processing unit 112 corrects the decoded data based on the decoded data and quantization parameters.

[0093] In step S21, the input interface 51 receives the bitstream transmitted from the output interface 32.

[0094] In step S22, the decoder 111 decodes the bitstream and reconstructs the decoded data. Specifically, the decoder 111 expands the bitstream to obtain the encoded data and quantization parameters, decodes the encoded data, and reconstructs the decoded data.

[0095] In step S23, the correction processing unit 112 estimates the maximum possible amount of degradation (maximum degradation amount X) that may occur due to compression coding, based on the quantization parameters. For example, the correction processing unit 112 has a table or function that shows the relationship between the quantization parameters, the bit depth of the original data, and the maximum degradation amount X. The correction processing unit 112 estimates the maximum degradation amount X by referring to the table based on the combination of quantization parameters and the bit depth of the original data.

[0096] Figure 12 shows an example of a table illustrating the relationship between quantization parameters and the maximum degradation amount X.

[0097] In the example in Figure 12, when the quantization parameter Qp is 0, the maximum degradation amount X is 0, and when the quantization parameter Qp is 1, the maximum degradation amount X is 2. Also, when the quantization parameter Qp is 2, the maximum degradation amount X is 4, and when the quantization parameter Qp is 3, the maximum degradation amount X is 8.

[0098] The correction processing unit 112 has, for example, a table or function that shows the relationship between such quantization parameter Qp and the maximum degradation amount X for each bit depth of the original data, and can estimate the maximum degradation amount X by referring to the table corresponding to the bit depth of the original data.

[0099] Returning to Figure 11, in step S24, the correction processing unit 112 defines a threshold value b. Here, the threshold value b is, for example, the saturation value (2 ビット深度 It is defined as the value obtained by subtracting the maximum degradation amount X from (-1). In other words, the threshold b is a value defined based on the quantization parameter Qp. After the threshold b is defined, the correction processing unit 112 sets the pixels to be corrected from all the pixels of the decoded data.

[0100] The processing in steps S25 to S29 is the same as the processing in steps S4 to S8 in Figure 9. That is, if the pixel value a is greater than or equal to the threshold b, the saturation value is used as the corrected pixel value of the pixel to be corrected, and if the pixel value a is less than the threshold b, the pixel value a is used as the corrected pixel value of the pixel to be corrected. This correction process is performed for all pixels of the decoded data.

[0101] Next, with reference to the flowchart in Figure 13, a third process performed by the input I / F 51 having the configuration shown in Figure 6 will be described. In Figure 13, the process performed by the input I / F 51 when the correction processing unit 112 corrects the decoded data based on the decoded data, quantization parameters, and prediction mode information will be described.

[0102] In step S41, the input interface 51 receives the bitstream transmitted from the output interface 32.

[0103] In step S42, the decoder 111 decodes the bitstream and reconstructs the decoded data. Specifically, the decoder 111 expands the bitstream to obtain encoded data, quantization parameters, and prediction mode information, decodes the encoded data, and reconstructs the decoded data. Subsequently, the correction processing unit 112 sets the pixels to be corrected from all the pixels of the decoded data.

[0104] In step S43, the correction processing unit 112 inputs prediction mode information, quantization parameters, and decoded data for the pixels to be corrected into an inference unit generated by machine learning such as deep learning. The correction processing unit 112 has an inference unit that estimates whether the pixels to be corrected in the decoded data correspond to the saturated pixels in the original data, and can identify the corresponding pixels of saturated pixels in the decoded data based on the inference results obtained by inputting this data into the inference unit.

[0105] In step S44, the correction processing unit 112 determines, by the inference unit, whether or not the pixel to be corrected is estimated to be a pixel (corresponding pixel) corresponding to a saturated pixel in the original data.

[0106] If it is estimated that the pixel to be corrected corresponds to the saturated pixel of the original data, in step S45, the correction processing unit 112 performs a correction to set the saturated value to the corrected pixel value of the pixel to be corrected.

[0107] On the other hand, if it is estimated that the pixel to be corrected is not a pixel corresponding to a saturated pixel in the original data, in step S46, the correction processing unit 112 performs a correction in which the pixel value a is used as is and becomes the corrected pixel value of the pixel to be corrected.

[0108] Furthermore, the correction processing unit 112 can identify the corresponding pixels of saturated pixels in the decoded data using a rule-based method, such as a table or function, rather than using an inference unit to identify the corresponding pixels of saturated pixels in the decoded data.

[0109] After the processing in step S45 or step S46 is performed, in step S47, the correction processing unit 112 determines whether or not correction processing has been performed on all pixels of the decoded data.

[0110] If it is determined in step S47 that some pixels have not been corrected, the correction processing unit 112 sets the next pixel to be corrected. After the pixel to be corrected is set, the process returns to step S43, and the subsequent processing is repeated.

[0111] On the other hand, if it is determined in step S8 that correction processing has been performed on all pixels, the process ends.

[0112] Figure 14 shows examples of the number of occurrences of pixel values ​​in the original data, the decoded data before correction, and the decoded data after correction.

[0113] As shown by the dashed line in the upper part of Figure 14, in the original data, the saturation value appears overwhelmingly more often among the pixel values ​​near the saturation value.

[0114] As shown by the dashed line in the middle section of Figure 14, in the decoded data before correction, the number of occurrences of saturation values ​​is less than in the original data, and the number of occurrences of pixel values ​​near the saturation value, excluding the saturation value, is more than in the original data.

[0115] As shown by the dashed line in the lower part of Figure 14, in the corrected decoded data, the number of occurrences of the saturation value is approximately the same as in the original data, and the number of occurrences of pixel values ​​near the saturation value, excluding the saturation value itself, is less than in the original data and the uncorrected decoded data.

[0116] <3. Modified Examples> - Example with a different encoding processing order Figure 15 is a block diagram showing a first modified example of the configuration of the transmission I / F connecting the image sensor 11 and the companion chip 12. In Figure 15, components identical to those in Figure 6 are denoted by the same reference numerals. Repetitive explanations are omitted as appropriate.

[0117] The output interface 32 in Figure 15 differs from the output interface 32 in Figure 6 in that it has an encoder 201 instead of an encoder 101. Also, the input interface 51 in Figure 15 differs from the input interface 51 in Figure 6 in that it has a decoder 211 instead of a decoder 111.

[0118] The encoder 201 performs intra-encoding on the image data of the current frame to generate a bitstream. As will be described in detail later, the encoder 201 performs intra-encoding by quantizing the image data before performing intra-prediction.

[0119] The decoder 211 performs intra-decoding to reconstruct the decoded data of the current frame. As will be described in detail later, the decoder 211 performs intra-decoding by performing intra-prediction before dequantizing the quantized data.

[0120] Figure 16 is a block diagram showing a detailed configuration example of the encoder 201.

[0121] The encoder 201 applies compression encoding to each pixel block into which the input raw data has been divided. Alternatively, compression encoding may be applied to each pixel of the raw data.

[0122] As shown in Figure 16, the encoder 201 is composed of a quantization unit 231, an encoding control unit 232, an intra predictor 233, a subtractor 234, an encoding unit 235, an adder 236, an inverse quantization unit 237, a reference buffer 238, a quantization unit 239, and a stream generation unit 240.

[0123] The quantization unit 231 quantizes the pixel block to be compressed using quantization parameters suitable for that pixel block, and supplies the resulting quantized data to the subtractor 234. The quantization unit 231 also supplies the quantization parameters used to quantize the pixel block to the encoding control unit 232 and the stream generation unit 240.

[0124] The coding control unit 232 determines the prediction mode to be applied when generating prediction image data, based on the quantization parameters supplied from the quantization unit 231 and the reference image data (reference quantized data) quantized by the quantization unit 239. The coding control unit 232 supplies the prediction mode information to the intra predictor 233 and the stream generation unit 240.

[0125] The intra predictor 233 performs intra prediction, generating predicted image data (predicted quantized data) quantized from the reference quantized data supplied from the quantization unit 239 in a prediction mode determined by the coding control unit 232. Here, the reference image data is image data composed of compressed pixels (pixels that have undergone quantization and conversion to predicted residuals once) surrounding the pixel block that is subject to compression coding, from among the pixels of the entire original data. The intra predictor 233 supplies the predicted quantized data to the subtractor 234 and the adder 236.

[0126] The subtractor 234 subtracts the predicted quantized data from the quantized data supplied by the quantization unit 231 to generate a predicted residual, which is then supplied to the encoding unit 235 and the adder 236.

[0127] The encoding unit 235 entropy encodes the predicted residual supplied from the subtractor 234 and supplies the resulting encoded data to the stream generation unit 240.

[0128] The adder 236 adds the predicted quantized data supplied from the intra predictor 233 and the predicted residual supplied from the subtractor 234 to reconstruct the quantized data in which the pixel blocks to be compressed and encoded have been quantized. The adder 236 supplies the reconstructed quantized data to the inverse quantization unit 237.

[0129] The inverse quantization unit 237 inversely quantizes the quantized data supplied from the adder 236 and supplies the resulting data of the pixel block to be compressed and encoded to the reference buffer 238.

[0130] The reference buffer 238 holds the data supplied from the inverse quantization unit 237 and supplies the reference image data to the quantization unit 239 in response to requests from the coding control unit 232 and the intra predictor 233.

[0131] The quantization unit 239 quantizes the reference image data supplied from the reference buffer 238 and supplies the resulting reference quantized data to the encoding control unit 232 and the intra predictor 233.

[0132] The stream generation unit 240 combines the quantization parameters supplied from the quantization unit 231, the prediction mode information supplied from the coding control unit 232, and the coded data supplied from the coding unit 235 to generate a bitstream.

[0133] Figure 17 is a block diagram showing a detailed configuration example of the decoder 211.

[0134] As shown in Figure 17, the decoder 211 is composed of an extension unit 251, a decoding unit 252, an intra predictor 253, an adder 254, an inverse quantization unit 255, a reference buffer 256, and a quantization unit 257.

[0135] The decompression unit 251 decompresses the bitstream to obtain quantization parameters, prediction mode information, and encoded data. The decompression unit 251 supplies the quantization parameters to the inverse quantization unit 255, the quantization unit 257, and the correction processing unit 112. The decompression unit 251 supplies the prediction mode information to the intra predictor 253 and the correction processing unit 112. The decompression unit 251 supplies the encoded data to the decoding unit 252.

[0136] The decoding unit 252 entropy-decodes the encoded data supplied from the decompression unit 251 and supplies the resulting quantized prediction residual to the adder 254.

[0137] The intra predictor 253 performs intra prediction, generating predicted image data (predicted quantized data) from reference image data (reference quantized data) quantized by the quantization unit 257 in the prediction mode indicated by the prediction mode information supplied from the decompression unit 251. Here, the reference image data is image data composed of decoded pixels surrounding the pixel block to be decoded, among the pixels of the entire decoded data. The intra predictor 253 supplies the predicted quantized data to the adder 254.

[0138] The adder 254 adds the quantized prediction residual supplied from the decoding unit 252 and the prediction quantized data supplied from the intra predictor 253 to reconstruct the quantized data in which the pixel block to be decoded has been quantized. The adder 254 supplies the reconstructed quantized data to the inverse quantization unit 255.

[0139] The inverse quantization unit 255 uses the quantization parameters supplied from the decompression unit 251 to inverse quantize the quantized data supplied from the adder 254, and supplies the resulting pixel blocks as decoded data to the reference buffer 256 and the correction processing unit 112.

[0140] The reference buffer 256 holds the decoded data supplied from the inverse quantization unit 255 and supplies the reference image data to the quantization unit 257 in response to a request from the intra predictor 253.

[0141] The quantization unit 257 uses the quantization parameters supplied from the decompression unit 251 to quantize the reference image data supplied from the reference buffer 256, and supplies the resulting reference quantized data to the intra predictor 253.

[0142] A fourth process performed by an input I / F 51 having the configuration shown in Figure 6 or Figure 15 will be described, referring to the flowchart in Example Figure 18, which identifies the corresponding pixels of the maximum and minimum pixels in the decoded data based on the quantized decoded data. In Figure 18, the process performed by the input I / F 51 when the correction processing unit 112 corrects the decoded data based on the decoded data and quantization parameters will be described.

[0143] In step S61, the input interface 51 receives the bitstream transmitted from the output interface 32.

[0144] In step S62, the decoder 111 decodes the bitstream and reconstructs the decoded data.

[0145] In step S63, the correction processing unit 112 defines a threshold value b based on the quantization parameters. Here, the threshold value b is, for example, the saturation value (2 ビット深度 It is defined as the integer part of the quotient when -1) is divided by the quantization parameter Qp. After the threshold b is defined, the correction processing unit 112 sets the pixels to be corrected from all the pixels of the decoded data.

[0146] In step S64, the correction processing unit 112 obtains the pixel value a of the pixel to be corrected.

[0147] In step S65, the correction processing unit 112 defines the quantized pixel value a'. Here, the quantized pixel value a' is defined, for example, as the integer part of the quotient obtained when the pixel value a is divided by the quantization parameter Qp.

[0148] In step S66, the correction processing unit 112 determines whether the quantized pixel value a' matches the threshold value b.

[0149] If it is determined in step S66 that the quantized pixel value a' matches the threshold b, the correction processing unit 112 identifies the pixel to be corrected as the corresponding pixel of the saturated pixel. Then, in step S67, the correction processing unit 112 performs a correction to set the saturation value as the corrected pixel value of the pixel to be corrected.

[0150] On the other hand, if it is determined in step S66 that the quantized pixel value a' and the threshold b do not match, the correction processing unit 112 identifies the pixel to be corrected as a pixel that is not the corresponding pixel of the saturated pixel. Then, in step S68, the correction processing unit 112 performs a correction, using the pixel value a as is to be corrected as the corrected pixel value of the pixel to be corrected.

[0151] After the processing in step S67 or step S68 is performed, in step S69, the correction processing unit 112 determines whether or not correction processing has been performed on all pixels of the decoded data.

[0152] If it is determined in step S69 that some pixels have not been corrected, the correction processing unit 112 sets the next pixel to be corrected. After the pixel to be corrected is set, the process returns to step S64, and the subsequent processing is repeated.

[0153] On the other hand, if it is determined in step S69 that correction processing has been performed on all pixels, the process ends.

[0154] Figure 19 is a diagram illustrating the details of the quantized pixel value a' and threshold b. In the example in Figure 19, the quantization parameter Qp is, for example, 2 4The pixel value a is represented by 10 bits of data. The quantization parameter Qp does not necessarily have to be a power of 2 with a positive integer exponent, but can be any real number.

[0155] Since the quantized pixel value a' is the integer part obtained by dividing the pixel value a by the quantization parameter Qp, as shown in the upper part of Figure 19, the data of the quantized pixel value a' can be said to be an extraction of the upper 6 bits of the data of pixel value a. Since the degradation caused by compression coding (especially quantization) is contained in the lower 4 bits of the data of pixel value a, the data of the quantized pixel value a' does not contain the degradation caused by compression coding.

[0156] Furthermore, since threshold b is the integer part obtained by dividing the saturation value by the quantization parameter Qp, as shown in the lower part of Figure 19, the threshold b data can be said to be an extraction of the upper 6 bits of the saturation value data. Since the degradation caused by compression coding (especially quantization) is contained in the lower 4 bits of the saturation value data, the threshold b data does not include the degradation caused by compression coding.

[0157] In this way, by quantizing the pixel value a, the correction processing unit 112 can identify the corresponding pixel of the saturated pixel in the decoded data using data that does not contain the degradation caused by compression.

[0158] Furthermore, if prediction mode information is input to the correction processing unit 112 in addition to the decoded data and quantization parameters, the correction processing unit 112 can, for example, identify the corresponding pixels of saturated pixels in the decoded data by inputting the quantized decoded data and prediction mode information to the inference unit.

[0159] Figure 20 is a block diagram showing a second modified configuration of the transmission interface connecting the image sensor 11 and the companion chip 12. In Figure 20, components identical to those in Figure 6 are denoted by the same reference numerals. Repetitive explanations are omitted as appropriate.

[0160] The output interface 32 in Figure 20 is the same as the output interface 32 in Figure 6. The input interface 51 in Figure 20 differs from the input interface 51 in Figure 6 in that the decoder 111 has a correction processing unit 112.

[0161] As shown in Figure 20, the decoder 111 may be configured to include a correction processing unit 112 and a configuration related to bitstream decoding (such as the extension unit 151 and decoding unit 152 in Figure 8). Similarly, the decoder 211 in Figure 15 may also be configured to include a correction processing unit 112 and a configuration related to bitstream decoding (such as the extension unit 251 and decoding unit 252 in Figure 17).

[0162] Figure 21 is a block diagram showing a third modified example of the configuration of the transmission interface connecting the image sensor 11 and the companion chip 12. In Figure 21, components identical to those in Figure 6 are denoted by the same reference numerals. Repetitive explanations are omitted as appropriate.

[0163] The output interface 32 in Figure 21 is the same as the output interface 32 in Figure 6. The input interface 51 in Figure 21 differs from the input interface 51 in Figure 6 in that it is equipped with an inference unit 301.

[0164] The decoder 111 supplies the decoded data to the correction processing unit 112 and the inference unit 301.

[0165] The inference unit 301 takes the decoded data supplied from the decoder 111 as input, estimates whether each pixel of the decoded data corresponds to the maximum or minimum pixel of the original data, and supplies the estimation result (inference result) to the correction processing unit 112.

[0166] The correction processing unit 112 identifies the corresponding pixels of the maximum and minimum pixels in the decoded data based on the inference results from the inference unit 301, and corrects the pixel values ​​of the corresponding pixels of the maximum and minimum pixels in the decoded data.

[0167] Furthermore, the inference unit 301 may be provided with input not only of the decoded data, but also of quantization parameters and predicted mode information.

[0168] Thus, an inference unit that determines whether each pixel of the decoded data corresponds to the maximum or minimum pixel of the original data may be implemented outside the correction processing unit 112.

[0169] Figure 22 is a block diagram showing a fourth modified example of the configuration of the transmission interface connecting the image sensor 11 and the companion chip 12. In Figure 22, components identical to those in Figure 6 are denoted by the same reference numerals. Repetitive explanations are omitted as appropriate.

[0170] The output I / F 32 in Figure 22 differs from the output I / F 32 in Figure 6 in that it acquires imaging parameters. The input I / F 51 in Figure 22 differs from the input I / F 51 in Figure 6 in that imaging parameters are supplied from the decoder 111 to the correction processing unit 112.

[0171] The output interface 32 acquires imaging parameters related to image acquisition (imaging) from each component constituting the image sensor 11, such as the pixels 31. These imaging parameters include parameters related to the focus position, aperture, image stabilization, and exposure.

[0172] The encoder 101 combines the encoded data, quantization parameters, prediction mode information, and imaging parameters to generate a bitstream.

[0173] The decoder 111 decodes the bitstream transmitted from the output I / F 32 to obtain decoded data, quantization parameters, prediction mode information, and imaging parameters, and supplies them to the correction processing unit 112.

[0174] The correction processing unit 112 corrects the degradation that occurred in the decoded data due to compression coding, based on, for example, the quantization parameters, prediction mode information, and imaging parameters supplied from the decoder 111.

[0175] Figure 23 is a block diagram showing a fifth modified configuration of the transmission interface connecting the image sensor 11 and the companion chip 12. In Figure 23, components identical to those in Figure 6 are denoted by the same reference numerals. Repetitive explanations are omitted as appropriate.

[0176] The output interface 32 in Figure 23 is the same as the output interface 32 in Figure 6. The input interface 51 in Figure 23 differs from the input interface 51 in Figure 6 in that an image processing device 311 is provided.

[0177] The decoder 111 decodes the bitstream transmitted from the output I / F 32 to generate decoded data, which is then supplied to the correction processing unit 112 and the image processing device 311.

[0178] The image processing device 311 sets processing parameters to control the correction processing performed by the correction processing unit 112 based on the decoded data supplied from the decoder 111. The processing parameters include information indicating a threshold b, information indicating the positions of the corresponding pixels for the maximum and minimum pixels in the decoded data, and so on.

[0179] The correction processing unit 112 corrects the degradation that occurred in the decoded data due to compression encoding based on the processing parameters supplied from the image processing device 311.

[0180] • Description of a computer to which this technology is applied: The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up the software are installed on the computer. Here, the term "computer" includes computers built into dedicated hardware, as well as general-purpose personal computers, for example, that can perform various functions by installing various programs.

[0181] Figure 24 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above using a program.

[0182] In a computer, the processing circuit 501, ROM (Read Only Memory) 502, and RAM (Random Access Memory) 503 are interconnected by a bus 504.

[0183] An input / output interface 505 is further connected to the bus 504. An input unit 506, an output unit 507, a storage unit 508, a communication unit 509, and a drive 510 are connected to the input / output interface 505.

[0184] The input unit 506 may include physical or virtual operating means that the user operates to input information, such as a keyboard, mouse, or touch panel, as well as means that the user inputs information through voice, eye gaze, etc. Furthermore, the input unit 506 may include sensors for inputting various physical quantities to the computer. For example, the input unit 506 may include sensors that acquire physical quantities such as light (including infrared light other than visible light) or sound, such as a camera or microphone. Also, for example, the input unit 506 may include sensors that acquire other physical quantities such as temperature, moisture content, acceleration, distance, etc. The output unit 507 may include means that present information to the user by stimulating the user's perception, such as a display, speaker, or haptic device. The storage unit 508 is composed of a hard disk, non-volatile or volatile memory, etc., and stores various information (including programs). The communication unit 509 is a network interface, etc., and performs wired or wireless communication with the outside. The drive 510 drives removable media 511 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0185] The processing circuit 501 includes a processor that executes programs such as a CPU (Central Processing Unit) and a DSP (Digital Signal Processor). The processing circuit 501 (its processor) performs the series of processes described above by loading the program stored in the storage unit 508 into the RAM 503 via the input / output interface 505 and the bus 504 and executing it. The processing circuit 501 can output the processing results of the series of processes from the output unit 507 via the bus 504 and the input / output interface 505 as needed. The processing circuit 501 can also store the processing results in the storage unit 508 or transmit them from the communication unit 509.

[0186] The program executed by the computer (processing circuit 501) can be provided by recording it on a removable medium 511, such as a package medium. The program can also be provided via wired or wireless transmission media, such as a local area network, the internet, or digital satellite broadcasting.

[0187] In a computer, a program can be installed in the storage unit 508 via the input / output interface 505 by inserting the removable media 511 into the drive 510. Alternatively, a program can be received by the communication unit 509 from another device, such as a server, via a wired or wireless transmission medium, and installed in the storage unit 508. Furthermore, programs can be pre-installed in the ROM 502 or the storage unit 508.

[0188] The programs executed by the computer may be programs that are processed chronologically in the order described herein, or they may be programs that are processed in parallel or at necessary times, such as when a call is made.

[0189] The processes that a computer performs according to a program do not necessarily have to follow the order described in the flowchart. In other words, the processes that a computer performs according to a program include processes that are executed in parallel or individually (e.g., parallel processing and object-based processing).

[0190] The program may be processed by a single computer (processor), or it may be processed in a distributed manner by multiple computers. Furthermore, the program may be transferred to a remote computer and executed there.

[0191] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all components are located in the same enclosure. Therefore, multiple devices housed in separate enclosures and connected via a network, and a single device containing multiple modules within a single enclosure, are both considered systems.

[0192] The effects described herein are illustrative and not limited to those described herein, and other effects may also occur.

[0193] The embodiments of this technology are not limited to those described above, and various modifications are possible without departing from the spirit of this technology.

[0194] Furthermore, each step described in the flowchart above can be performed by a single device, or it can be divided and performed by multiple devices.

[0195] Furthermore, if a single step includes multiple processes, those processes can be executed by a single device or shared among multiple devices.

[0196] <Examples of configuration combinations> This technology can also be configured as follows:

[0197] (1) An information processing device comprising a correction processing unit that corrects the pixel value of a corresponding pixel, which is estimated to correspond to the maximum or minimum pixel of the image data, to the maximum or minimum value in decoded data obtained by decoded bitstream, which is compressed and encoded bitstream, which includes maximum and minimum pixels, which are pixels for which a maximum or minimum value has been set as the pixel value. (2) The information processing device according to (1), wherein the correction processing unit identifies the corresponding pixel of the maximum or minimum pixel in the decoded data by comparing the pixel value of each pixel in the decoded data with a pre-prepared threshold. (3) The information processing device according to (1), wherein the correction processing unit identifies the corresponding pixel of the maximum or minimum pixel in the decoded data based on the quantization parameters used for the compression encoding of the image data. (4) The information processing device according to (3), wherein the correction processing unit identifies the corresponding pixel of the maximum or minimum pixel in the decoded data by comparing the pixel value of each pixel in the decoded data with a threshold defined based on the quantization parameters. (5) The information processing apparatus according to (3), wherein the correction processing unit identifies the corresponding pixels of the maximum and minimum pixels in the decoded data by comparing the quantized pixel values ​​of each pixel in the decoded data using the quantization parameters with a threshold value defined based on the quantization parameters. (6) The information processing apparatus according to (1), wherein the correction processing unit identifies the corresponding pixels of the maximum and minimum pixels in the decoded data based on the quantization parameters used for the compression encoding of the image data and prediction mode information indicating the prediction mode applied when the image data was compressed and encoded. (7) The information processing apparatus according to (6), wherein the correction processing unit identifies the corresponding pixels of the maximum and minimum pixels in the decoded data based on the inference result obtained by inputting the quantization parameters, the prediction mode information, and the decoded data into an inference unit generated by machine learning. (8) The information processing apparatus according to any one of (1) to (7), further comprising a decoding unit that decodes the bitstream and generates the decoded data.(9) The information processing apparatus according to (8), wherein the decoding unit generates the decoded data by adding the predicted image data and the predicted residual obtained by inverse quantization of the quantized data acquired from the bitstream. (10) The information processing apparatus according to (8), wherein the decoding unit generates the decoded data by inverse quantization of the data obtained by adding the quantized predicted image data and the quantized predicted residual obtained from the bitstream. (11) An information processing method comprising correcting the pixel value of a corresponding pixel, which is a pixel estimated to correspond to the maximum or minimum pixel of the image data, to the maximum or minimum value in decoded data obtained by decoding a bitstream in which image data including a maximum or minimum pixel, which is a pixel for which a maximum or minimum value is set as a pixel value, is compressed and encoded. (12) An information processing system comprising: a first information processing device comprising: an encoding unit that compresses and encodes image data including maximum and minimum pixels, which are pixels for which a maximum or minimum value is set as a pixel value, to generate a bitstream; a decoding unit that decodes the bitstream to generate decoded data; and a second information processing device that corrects the pixel value of a corresponding pixel, which is a pixel estimated to correspond to the maximum and minimum pixels of the image data, to the maximum or minimum value in the decoded data. (13) The information processing system according to (12), wherein the correction processing unit identifies the corresponding pixel of the maximum and minimum pixels in the decoded data by comparing the pixel value of each pixel in the decoded data with a pre-prepared threshold. (14) The information processing system according to (12), wherein the correction processing unit identifies the corresponding pixel of the maximum and minimum pixels in the decoded data based on quantization parameters used for compressing and encoding the image data. (15) The information processing system according to (14), wherein the correction processing unit identifies the corresponding pixel of the maximum and minimum pixels in the decoded data by comparing the pixel value of each pixel in the decoded data with a threshold defined based on the quantization parameters.(16) The information processing system according to (14), wherein the correction processing unit identifies the corresponding pixels of the maximum and minimum pixels in the decoded data by comparing the quantized pixel values ​​of each pixel in the decoded data using the quantization parameters with a threshold value defined based on the quantization parameters. (17) The information processing system according to (12), wherein the correction processing unit identifies the corresponding pixels of the maximum and minimum pixels in the decoded data based on the quantization parameters used for the compression encoding of the image data and prediction mode information indicating the prediction mode applied when the image data was compressed and encoded. (18) The information processing system according to (17), wherein the correction processing unit identifies the corresponding pixels of the maximum and minimum pixels in the decoded data based on the inference result obtained by inputting the quantization parameters, the prediction mode information, and the decoded data into an inferencer generated by machine learning. (19) The information processing system according to any one of (12) to (18), wherein the decoding unit generates the decoded data by adding the predicted image data and the prediction residual obtained by inverse quantization of the quantized data acquired from the bitstream. (20) The information processing system according to any one of (12) to (18) above, wherein the decoding unit generates the decoded data by dequantizing the data obtained by adding the quantized predicted image data and the quantized predicted residual obtained from the bitstream.

[0198] 11 Image sensor, 12 Companion chip, 13 DRAM, 31 Pixel, 32 Output I / F, 51 Input I / F, 52 Image processing unit, 53 Stabilization processing unit, 54 Codec processing unit, 55 to 57 Input / Output I / F, 58 Bus, 71 Preprocessing unit, 72 3DNR processing unit, 73 Development processing unit, 101 Encoder, 111 Decoder, 112 Correction processing unit, 201 Encoder, 211 Decoder, 301 Inference unit, 311 Image processing unit

Claims

1. An information processing device comprising a correction processing unit that corrects the pixel values ​​of corresponding pixels, which are estimated to correspond to the maximum or minimum pixels of the image data, to the maximum or minimum value in decoded data obtained by decoding a bitstream in which image data including maximum or minimum pixels, which are pixels for which a maximum or minimum value has been set as a pixel value.

2. The information processing apparatus according to claim 1, wherein the correction processing unit identifies the corresponding pixels of the maximum and minimum pixels in the decoded data by comparing the pixel value of each pixel in the decoded data with a pre-prepared threshold.

3. The information processing apparatus according to claim 1, wherein the correction processing unit identifies the corresponding pixels of the maximum and minimum pixels in the decoded data based on the quantization parameters used for the compression encoding of the image data.

4. The information processing apparatus according to claim 3, wherein the correction processing unit identifies the corresponding pixels of the maximum and minimum pixels in the decoded data by comparing the pixel value of each pixel in the decoded data with a threshold value defined based on the quantization parameter.

5. The information processing apparatus according to claim 3, wherein the correction processing unit identifies the corresponding pixels of the maximum and minimum pixels in the decoded data by comparing the quantized pixel value of each pixel in the decoded data using the quantization parameter with a threshold value defined based on the quantization parameter.

6. The information processing apparatus according to claim 1, wherein the correction processing unit identifies the corresponding pixels of the maximum and minimum pixels in the decoded data based on the quantization parameters used for the compression encoding of the image data and prediction mode information indicating the prediction mode applied when the image data was compressed and encoded.

7. The information processing apparatus according to claim 6, wherein the correction processing unit identifies the corresponding pixels of the maximum and minimum pixels in the decoded data based on the inference result obtained by inputting the quantization parameters, the prediction mode information, and the decoded data into an inference unit generated by machine learning.

8. The information processing apparatus according to claim 1, further comprising a decoding unit that decodes the bitstream and generates the decoded data.

9. The information processing apparatus according to claim 8, wherein the decoding unit generates the decoded data by adding the predicted image data and the predicted residual obtained by inverse quantization of the quantized data acquired from the bitstream.

10. The information processing apparatus according to claim 8, wherein the decoding unit generates the decoded data by dequantizing the data obtained by adding the quantized predicted image data and the quantized predicted residual obtained from the bitstream.

11. An information processing method that includes correcting the pixel values ​​of corresponding pixels, which are estimated to correspond to the maximum or minimum pixels of the image data, to the maximum or minimum value in decoded data obtained by decoding a compressed and encoded bitstream of image data including maximum or minimum pixels, which are pixels for which a maximum or minimum value has been set as a pixel value.

12. An information processing system comprising: a first information processing device comprising: an encoding unit that compresses and encodes image data including maximum and minimum pixels, which are pixels for which a maximum or minimum value is set as a pixel value, to generate a bitstream; a decoding unit that decodes the bitstream to generate decoded data; and a correction processing unit that corrects the pixel values ​​of corresponding pixels, which are pixels estimated to correspond to the maximum and minimum pixels of the image data, to the maximum or minimum value in the decoded data.

13. The information processing system according to claim 12, wherein the correction processing unit identifies the corresponding pixels of the maximum and minimum pixels in the decoded data by comparing the pixel value of each pixel in the decoded data with a pre-prepared threshold.

14. The information processing system according to claim 12, wherein the correction processing unit identifies the corresponding pixels of the maximum and minimum pixels in the decoded data based on the quantization parameters used for the compression encoding of the image data.

15. The information processing system according to claim 14, wherein the correction processing unit identifies the corresponding pixels of the maximum and minimum pixels in the decoded data by comparing the pixel value of each pixel in the decoded data with a threshold value defined based on the quantization parameter.

16. The information processing system according to claim 14, wherein the correction processing unit identifies the corresponding pixels of the maximum and minimum pixels in the decoded data by comparing the quantized pixel value of each pixel in the decoded data using the quantization parameter with a threshold value defined based on the quantization parameter.

17. The information processing system according to claim 12, wherein the correction processing unit identifies the corresponding pixels of the maximum and minimum pixels in the decoded data based on the quantization parameters used for the compression encoding of the image data and prediction mode information indicating the prediction mode applied when the image data was compressed and encoded.

18. The information processing system according to claim 17, wherein the correction processing unit identifies the corresponding pixels of the maximum and minimum pixels in the decoded data based on the inference result obtained by inputting the quantization parameters, the prediction mode information, and the decoded data into an inference unit generated by machine learning.

19. The information processing system according to claim 12, wherein the decoding unit generates the decoded data by adding the predicted image data and the predicted residual obtained by inverse quantization of the quantized data acquired from the bitstream.

20. The information processing system according to claim 12, wherein the decoding unit generates the decoded data by dequantizing the data obtained by adding the quantized predicted image data and the quantized predicted residual obtained from the bitstream.

Images