Encoder, decoder, and optical sensor

Abstract

The present technology pertains to an encoder, a decoder, and an optical sensor that make it possible to compress, while suppressing deterioration in image quality, a captured image composed of pixel values acquired by a clocked detector (CLK-DET) method in which counting of photons incident on a pixel during a count period for which a maximum count value of the photons is determined is performed as many times as the number of count periods in an exposure time, and a pixel value is acquired on the basis of a count value obtained by counting the photons. The encoder encodes a captured image composed of pixel values acquired by the CLK-DET method, on the basis of an encoding parameter generated on the basis of a CLK-DET parameter of the CLK-DET method. The decoder decodes the encoded data obtained by the encoding. The present technology can be applied, for example, to compression of a captured image composed of pixel values acquired by the CLK-DET method.

Description

Symbolizer, decoder, and optical sensor

[0001] The present technology relates to a symbolizer, a decoder, and an optical sensor, and particularly relates to a symbolizer, a decoder, and an optical sensor that enable, for example, a photographed image to be compressed while suppressing deterioration in image quality.

[0002] For example, as a technique for improving color unevenness and color shift under low illuminance and high illuminance respectively, there is a technique of determining the illuminance of a subject and adjusting the quantization step in the case of low illuminance or high illuminance (see, for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2004-194121

[0004] [[ID=​​​​​​​​The encoder or optical sensor of this technology is an encoder that counts photons incident on a pixel during a predetermined counting period, for the number of periods in the exposure time that correspond to the counting period, and based on the count value obtained from the photons, encodes an image composed of the pixel values ​​obtained by the CLK-DET (clocked detector) method, based on encoding parameters generated based on the CLK-DET parameters of the CLK-DET method, or an optical sensor including such an encoder.

[0008] In the encoder and optical sensor of this technology, the maximum count value of photons is determined by counting the photons incident on the pixel during a predetermined counting period, for the number of periods in the exposure time. Based on the count value obtained from counting the photons, the pixel values ​​are acquired using a CLK-DET (clocked detector) method, and the captured image composed of the pixel values ​​acquired by the CLK-DET method is encoded based on encoding parameters generated based on the CLK-DET parameters of the CLK-DET method.

[0009] The decoder of this technology is a decoder that decodes encoded data encoded by an encoding method which encodes an image composed of the pixel values ​​acquired by a CLK-DET (clocked detector) based on encoding parameters generated based on CLK-DET parameters of a CLK-DET (clocked detector) method in which the maximum count value of photons is determined by counting the number of periods of the counting period in the exposure time, and the pixel values ​​are acquired based on the count value of the photons counted by the CLK-DET method.

[0010] In the decoder of this technology, the maximum number of photons is determined by counting the photons incident on the pixel during a predetermined counting period, for the number of periods in the exposure time. Based on the count value obtained from the photons, the pixel values ​​are acquired using a CLK-DET (clocked detector) method, and the encoded data is decoded using an encoding method that encodes the captured image composed of the pixel values ​​acquired by the CLK-DET method, based on encoding parameters generated based on the CLK-DET parameters of the CLK-DET method.

[0011] The encoder and decoder may each be an independent device, or they may be internal blocks constituting a single independent device. Furthermore, each of the encoder and decoder may be composed of multiple independent devices.

[0012] The encoder and decoder can be implemented using dedicated hardware, or they can be implemented by running a program on a computer.

[0013] The program can be provided by recording it on a recording (storage) medium or by transmitting it via a transmission medium.

[0014] This is a block diagram showing an example configuration of a signal processing system to which this technology can be applied. This is a cross-sectional view showing an example of the schematic configuration of pixels in an optical sensor 11. This is a diagram illustrating an example of calculating pixel values ​​in an optical sensor 11. This is a diagram illustrating another example of calculating pixel values ​​in an optical sensor 11. This is a diagram illustrating photon counting using the CLK-DET method. This is a diagram illustrating an overview of QBI signal processing by the QBI signal processing unit 12. This is a diagram illustrating the degradation of image quality due to encoding of captured images obtained using the CLK-DET method. This is a diagram showing an example configuration of an optical sensor system to which this technology is applied. This is a flowchart illustrating an example of the operation of the optical sensor system 30. This is a block diagram showing an example configuration of an encoding parameter generator 31. This is a diagram illustrating an example of calculating a CLK-DET probability distribution using a probability distribution calculator 61. This is a diagram illustrating an example of the standard deviation of the CLK-DET probability distribution calculated by a standard deviation calculator 62. This is a diagram illustrating an example of a method for determining the brightness threshold using a brightness criterion determiner 63. This is a block diagram showing an overview of the configuration of an encoder 51 of a first type that determines the overall brightness of the processing target area and selects the encoding method for the processing target area based on the overall brightness of the processing target area. This is a block diagram illustrating an overview of the configuration of a second type of encoder 51, which determines the brightness of each pixel in the area to be processed and selects an encoding method for each pixel based on the brightness of each pixel in the area to be processed. This is a block diagram illustrating an overview of the configuration of a third type of encoder 51, which selects an entropy code to entropically encode the area to be processed from a plurality of entropy codes based on the area to be processed and / or the surrounding area of ​​the area to be processed. This is a block diagram illustrating an overview of the configuration of a fourth type of encoder 51, which encodes the area to be processed based on a CLK-DET probability distribution. This is a block diagram illustrating a first configuration example of the encoder 51. This is a block diagram illustrating an example of the configuration of the brightness determination unit 122. This is a block diagram illustrating another configuration example of the brightness determination unit 122. This is a diagram illustrating an example of adjusting the brightness threshold based on the QBI parameter in the threshold adjustment unit 141. This is a block diagram illustrating an example of the configuration of the encoder core 123. This is a block diagram illustrating an example of the configuration of the reversible encoding unit 158. This is a diagram illustrating the generation of a code table for bright areas and a code table for dark areas included in the code information supplied to the table selection unit 171.This is a block diagram illustrating another configuration example of the reversible coding unit 158. This diagram illustrates an example of an encoded stream generated by the stream generation unit 159. This is a block diagram illustrating a first configuration example of the decoder 33. This is a block diagram illustrating an example of the configuration of the reversible decoding unit 202. This is a block diagram illustrating another configuration example of the reversible decoding unit 202. This is a block diagram illustrating a second configuration example of the encoder 51. This diagram illustrates an example of pixel brightness information generated by the brightness determination unit 231. This is a block diagram illustrating an example of the configuration example of the brightness encoder core 232 and the darkness encoder core 233. This is a block diagram illustrating an example of the configuration of the reversible coding unit 248. This diagram illustrates the generation of multiple code tables included in the code information supplied to the table selection unit 271. This is a block diagram illustrating another configuration example of the reversible coding unit 248. This is a block diagram illustrating an example of the configuration of the reversible coding unit 261. This is a block diagram illustrating another configuration example of the reversible coding unit 261. This is a diagram illustrating an example of an encoded stream generated by the stream coupling unit 234. This is a block diagram illustrating another example of an encoded stream generated by the stream coupling unit 234. This is a block diagram illustrating a second configuration example of the decoder 33. This is a block diagram showing an example configuration of the bright area decoder core 312 and the dark area decoder core 313. This is a block diagram showing an example configuration of the reversible decoding unit 322. This is a block diagram showing another example configuration of the reversible decoding unit 322. This is a block diagram showing an example configuration of the reversible decoding unit 332. This is a block diagram showing another example configuration of the reversible decoding unit 332. This is a diagram showing an example configuration of another embodiment of an optical sensor system to which this technology is applied. This is a diagram illustrating another example of how the brightness threshold is determined by the brightness reference determination unit 63. This is a block diagram showing a first modified configuration example of the encoder core 123. This is a block diagram showing a first modified configuration example of the decoder 33. This is a block diagram showing a second modified configuration example of the encoder core 123. This is a block diagram showing a second modified configuration example of the decoder 33. This is a block diagram showing another example configuration of the bright area encoder core 232. This is a block diagram showing an example configuration of the quantization unit 441. This is a diagram illustrating an example of adjusting the quantization width by the quantization width determination unit 452. This is a block diagram showing an example configuration of a signal processing system to which the optical sensor system 30 is applied. This is a block diagram showing an example configuration of another signal processing system to which the optical sensor system 30 is applied.This is a block diagram showing an example configuration of one embodiment of a computer to which this technology is applied.

[0015] <Examples of signal processing system configurations to which this technology can be applied>

[0016] Figure 1 is a block diagram showing an example configuration of a signal processing system to which this technology can be applied.

[0017] In Figure 1, the signal processing system 10 includes an optical sensor 11, a QBI (quanta burst imaging) signal processing unit 12, a development processing unit 13, a display unit 14, an encoding unit 15, and a storage medium 16.

[0018] The light sensor 11 detects light incident on it. By detecting light, the light sensor 11 captures an image, for example. For example, the light sensor 11 performs photoelectric conversion of the light incident on it and acquires the corresponding pixel value using the CLK-DET method. The light sensor 11 outputs an image composed of the pixel values ​​acquired using the CLK-DET method.

[0019] The optical sensor 11 can capture images with an appropriate exposure time, and can also perform burst imaging (BI) with a short exposure time. The optical sensor 11 can perform BI at high frame rates such as 480fps or 960fps, which are higher than 240fps (frames per second). For example, the optical sensor 11 outputs a 960fps image captured by BI. The 960fps image output by the optical sensor 11 is supplied to the QBI signal processing unit 12 and the encoding unit 15.

[0020] The QBI signal processing unit 12 performs QBI signal processing on the 960fps images captured by the BI. Each frame (one image) of the 960fps images captured by the BI is taken with a short exposure time, so if the lighting is insufficient, the image will be dim. In QBI signal processing, multiple such dim images are superimposed to generate a non-blur (no motion blur) image with a lower frame rate than 960fps, for example, a high dynamic range (DR) image at 60fps. The QBI signal processing unit 12 supplies the 60fps image generated by QBI signal processing to the development processing unit 13.

[0021] Furthermore, by performing BI in a well-lit environment with the light sensor 11, it is possible to obtain high-DR images at high frame rates such as 480fps or 960fps. Such high-DR images at high frame rates enable the realization of super slow motion.

[0022] The development processing unit 13 develops the RAW data from the QBI signal processing unit 12 as a 60fps image, and supplies the developed 60fps image to the display unit 14.

[0023] The display unit 14 displays a 60fps image from the development processing unit 13.

[0024] Meanwhile, the encoding unit 15 compresses the 960fps captured images taken by BI by encoding them using a predetermined encoding method, and supplies them to the storage medium 16 for storage. The captured images stored in the storage medium 16 are decoded and used as needed. For example, the captured images can be used as material for content such as super slow-motion images, or as training data for AI (artificial intelligence) models.

[0025] In addition to capturing images using BI, the optical sensor 11 can also capture images at so-called normal frame rates such as 60fps or 120fps. The encoding unit 15 can encode the captured images obtained by capturing at normal frame rates and store them in the storage medium 16.

[0026] Figure 2 is a cross-sectional view showing a schematic example of the pixel configuration of the light sensor 11.

[0027] The light sensor 11 has, for example, a plurality of pixels arranged in a two-dimensional plane. The pixels of the light sensor 11 are composed of SPADs (Single Photon Avalanche Diodes). In a pixel, a flat plate-shaped P-type diffusion layer and an N-type diffusion layer are connected to form a SPAD, which extends over almost the entire pixel area in a planar view. A voltage that generates avalanche multiplication is applied to the SPAD, and an avalanche multiplication region is formed by a depletion layer formed at the connection portion between the P-type diffusion layer and the N-type diffusion layer.

[0028] When even one photon enters a pixel, and electrons are generated in the SPAD in response to that photon, a large current flows due to avalanche multiplication. The light sensor 11 counts when the current exceeds a threshold as one photon, and the pixel value is calculated according to the count value of that photon.

[0029] Figure 3 illustrates an example of how pixel values ​​are calculated using the light sensor 11.

[0030] In this specification, "calculation" means not only obtaining numerical values ​​through calculation, but also deriving some kind of information by methods other than calculation (for example, using a lookup table).

[0031] Figure 3 shows pulses that indicate a current exceeding a threshold has flowed through a pixel (SPAD). When a subject with high illumination, photons are incident on the pixel at a high frequency, and therefore, the light sensor 11 counts a large number of photons (indicating that the current has exceeded the threshold) during a predetermined exposure time. On the other hand, when a subject with low illumination, photons are incident on the pixel at a low frequency, and therefore, the light sensor 11 counts a small number of photons during the exposure time.

[0032] In the light sensor 11, for example, the photon count value during exposure time is calculated as the pixel value. In this case, the pixel value increases with higher illumination and decreases with lower illumination.

[0033] Figure 4 illustrates another example of how to calculate pixel values ​​in the light sensor 11.

[0034] In Figure 3, all photons incident on a pixel are counted. Counting all photons incident on a pixel composed of SPADs results in enormous power consumption.

[0035] Therefore, the CLK-DET (Clocked Detector) method can be adopted as the method for calculating (acquiring) pixel values ​​in the light sensor 11.

[0036] In the CLK-DET method, the maximum number of photons is determined by counting the photons incident on the pixel during a predetermined counting period. This counting is performed for the number of counting periods in the exposure time, and the pixel value is obtained based on the count value of the photons. For example, the sum of the photon count values ​​for each counting period in the exposure time is obtained as the pixel value.

[0037] The counting period is the period during which photons are counted, and the maximum photon count value is predetermined. For example, if the maximum photon count value is set to 1, the number of photons that can be counted in one counting period is limited to 1. In this case, if one photon is incident during one counting period, the count value is 1, and even if two or more photons are incident, the count value remains 1.

[0038] In the following, we assume that the maximum photon count value is 1 during a single counting period. In Figure 4, the numbers within the rectangles representing the counting periods represent the photon count values ​​for that period. The same applies to the figures described later. When the maximum photon count value is 1, the count value for a counting period will be either 0 or 1.

[0039] One or more counting periods can be included in the exposure time. The number of counting periods included in the exposure time is also called the number of periods.

[0040] For example, if the exposure time includes Nmax counting periods, the photons will be counted in the range of 0 to Nmax.

[0041] In the CLK-DET method, even if two or more photons are incident during one count period, only one photon is counted. Further, when the exposure time is provided with a maximum number Nmax of count periods, even if more than Nmax photons are incident on the pixel during the exposure time, only Nmax photons are counted. Therefore, it can be said that the CLK-DET method is a counting method that intentionally causes photon count loss by introducing a count period in which the maximum count value of photons is determined. Further, it can be said that the CLK-DET method is a method of estimating the number of photons incident on a pixel by suppressing power consumption by counting a part of the photons incident on the pixel.

[0042] In the CLK-DET method, the count period and the number of periods are parameters of the CLK-DET method (CLK-DET parameters). The response characteristics (imaging characteristics) of the photosensor 11, that is, the photon count value with respect to the input of photons (light), and thus the output characteristics of the pixel value, are determined by the count period and the number of periods.

[0043] As shown in FIG. 4, as the CLK-DET parameters, one type of count period (count periods of the same length) and the number of periods, which is the number of the count periods during the exposure time, can be set. Further, as the CLK-DET parameters, a plurality of types of count periods (with different lengths) and the number of periods, which is the number of each of the plurality of types of count periods during the exposure time, can be set. FIG. 4 shows the case where one type of count period is set and the case where two types of count periods with different lengths are set.

[0044] FIG. 5 is a diagram for explaining photon counting by the CLK-DET method.

[0045] In FIG. 5, the relationship between the response characteristics (Sensor Response Curve) of the photosensor 11, that is, the photon rate (Input Photon Rate) at which photons are incident on the photosensor 11, and the photon count value (Count) by the CLK-DET method is shown.

[0046] Here, the photon count value may refer to the photon count value in one count period, or the sum of the photon count values in each count period during the exposure time. In this specification, in any case, except for particularly unclear cases, it is referred to as the photon count value (photon count value).

[0047] Also, the saturation of the photon count value means that the photon count value becomes 1 in all of the count periods during the exposure time.

[0048] When one long count period is set for the exposure time, for a low-illuminance subject, photons are counted to some extent, and for a subject with a certain level of high illuminance, the photon count value saturates. Therefore, as shown by curve L1, the response characteristic of the photosensor 11 is such that an effective count value can be obtained at low illuminance (low photon rate).

[0049] On the other hand, when one short count period is set for the exposure time, for a low-illuminance subject, photons are not counted very much, and photons are not counted unless the illuminance is above a certain level of high illuminance. Therefore, as shown by curve L2, the response characteristic of the photosensor 11 is such that an effective count value can be obtained at high illuminance (high photon rate).

[0050] Therefore, a long count period is suitable for counting (detecting light) photons with low illuminance (low photon rate), and a short count period is suitable for counting photons with high illuminance (high photon rate).

[0051] As explained in Figure 4, the CLK-DET method can employ multiple counting periods of different lengths. For example, it is possible to employ three different counting periods: a short z [ns (nanoseconds)] counting period, a medium y [ns] counting period, and a long x [ns] counting period. In this case, the response characteristics of the light sensor 11 will be such that effective count values ​​can be obtained over a wide illuminance range from low to high illuminance, as shown by curve L3. Therefore, a high dynamic range (DR) that provides appropriate gradation over a wide illuminance range can be achieved. The method for detecting photons that achieve high DR by employing multiple counting periods is described in International Publication No. 2024 / 203275.

[0052] Figure 6 is a diagram illustrating the overview of QBI signal processing by the QBI signal processing unit 12 in Figure 1.

[0053] Each frame of the 960fps image captured by the light sensor 11 using BI is captured with a short exposure time, resulting in a dim image if the lighting is insufficient. In QBI signal processing, such dim images are superimposed while motion estimation and alignment based on motion estimation, generating a non-blurred, high-DR image. In this way, QBI signal processing can convert noisy, dim images captured at high speed into high-quality images suitable for viewing.

[0054] The CLK-DET method allows for the acquisition of high-DR images, but high-DR images result in a very large amount of data. Furthermore, in order to obtain non-blurred images or super slow-motion images, it is necessary to capture images using BI with the optical sensor 11. In this case, the captured images will have a high frame rate, resulting in an even larger amount of data.

[0055] The transmission interface (IF) that outputs (transmits) the captured image, which has a very large amount of data, to the outside of the optical sensor 11 requires very powerful processing capabilities, such as a large transmission bandwidth. Therefore, in the signal processing system 10 of Figure 1, a transmission interface with very powerful processing capabilities is required between the optical sensor 11 and the QBI signal processing unit 12 and encoding unit 15, which are supplied with the captured image output by the optical sensor 11, resulting in a heavy load.

[0056] Furthermore, in the QBI signal processing unit 12, a very large amount of data, such as captured images, is temporarily stored in memory during QBI signal processing. However, accessing this memory also requires a large transmission bandwidth, resulting in a heavy load.

[0057] Furthermore, the encoding unit 15 encodes the captured image and stores it in the storage medium 16. However, with existing encoding methods, captured images with a very large amount of data may not be sufficiently compressed. In this case, the amount of data in the encoded captured image is large, requiring a large transmission bandwidth between the encoding unit 15 and the storage medium 16, which increases the load.

[0058] Furthermore, existing encoding methods can significantly degrade the image quality of images captured using the CLK-DET method.

[0059] Figure 7 illustrates the degradation of image quality due to encoding of captured images obtained using the CLK-DET method.

[0060] Figure 7 shows the response characteristics of the light sensor 11, that is, the relationship between the photon rate at which photons enter the light sensor 11, the photon count value obtained by the CLK-DET method, and consequently, the pixel value acquired by the CLK-DET method.

[0061] As explained in Figure 4, the CLK-DET method intentionally introduces photon count loss by introducing a counting period in which the maximum photon count value is predetermined. Therefore, as shown in Figure 7, a large portion of the photon count values ​​obtained by the CLK-DET method, i.e., the pixel values ​​acquired by the CLK-DET method, are assigned to lower photon rates rather than higher photon rates. Consequently, in an image captured using the CLK-DET method, more bits (gradation) (wider range) are assigned to dark areas rather than bright areas.

[0062] When an image captured using the CLK-DET method described above is encoded and compressed using an existing encoding method, the image quality deteriorates significantly. For example, existing encoding methods delete a lot of information (data) from the dark areas of the captured image, and encode the dark areas with very few bits. As a result, the image quality of the dark areas, which are allocated many bits in the captured image, deteriorates significantly.

[0063] Furthermore, if the captured image is encoded using an existing encoding method before QBI signal processing, the decoded image obtained by decoding the encoded image will have a lot of information removed from the dark areas, resulting in an image with significantly degraded image quality in the dark areas. Therefore, QBI signal processing using such decoded images will also result in an image generated by superimposing the decoded images, which will also have significantly degraded image quality in the dark areas.

[0064] Therefore, when compressing captured images composed of pixel values ​​acquired by the CLK-DET method, there is a need for a new encoding method suitable for such captured images, which can compress the captured image while suppressing degradation of image quality, and in particular, a new encoding method that can protect dark areas.

[0065] Therefore, this technology employs an encoding method that encodes a captured image composed of pixel values ​​acquired by the CLK-DET method, based on encoding parameters generated based on CLK-DET parameters. This encoding method allows for the compression of a captured image composed of pixel values ​​acquired by the CLK-DET method while suppressing degradation of image quality. In particular, it can efficiently reduce the amount of data in the captured image while suppressing the degradation of image quality in dark areas that occurs when a captured image composed of pixel values ​​acquired by the CLK-DET method is encoded using existing encoding methods.

[0066] As a result, for example, in the signal processing system 10 shown in Figure 1, the transmission bandwidth from the optical sensor 11 to the QBI signal processing unit 12 and the storage medium 16, and the transmission bandwidth between the QBI signal processing unit 12 and the memory that stores captured images, etc., can be reduced. This reduces the load on the optical sensor 11, the QBI signal processing unit 12, and the storage medium 16.

[0067] In the CLK-DET method, not all photons incident on the light sensor are necessarily counted. Furthermore, in the CLK-DET method, the response characteristics of the light sensor are optimized by a combination of one or more counting periods and the duration of each of these periods. That is, the relationship between the photon rate (the rate at which photons enter the light sensor), the photon count value, and ultimately the pixel value is optimized. Therefore, as shown in Figure 7, the response characteristics of the light sensor are not linear but curved; however, according to these response characteristics, the photon rate corresponds to the photon count value and the pixel value. Thus, while the photon rate represents the amount of photons incident on the light sensor, the corresponding photon count value and pixel value can also be used to represent the amount of photons incident on the light sensor. In the following, the photon count value or pixel value may be used instead of the photon rate to represent the amount of photons incident on the light sensor, as appropriate.

[0068] <One embodiment of an optical sensor system applying this technology>

[0069] Figure 8 shows an example configuration of one embodiment of an optical sensor system to which this technology is applied.

[0070] In Figure 8, the optical sensor system 30 includes an encoding parameter generator 31, an optical sensor 32, and a decoder 33.

[0071] The coding parameter generator 31 is supplied with CLK-DET parameters, namely the count period Tp and the number of periods Nmax. These CLK-DET parameters can be input by the user from an external device, for example.

[0072] The coding parameter generator 31 generates coding parameters for the coding performed by the encoder 51 (described later) based on the CLK-DET parameters, and supplies them to the encoder 51 and the decoder 33.

[0073] The encoding parameters include, for example, a brightness threshold, code information, and a CLK-DET probability distribution. The brightness threshold is a threshold used to determine the brightness of the processing target portion of the captured image. The code information is information about a reversible code, such as an entropy code, and is, for example, a code table that associates the bit sequence to be encoded with the entropy code resulting from the entropy encoding of that bit sequence. The CLK-DET probability distribution is the probability distribution of photons incident during the count period in the exposure time.

[0074] The light sensor 32 detects light incident upon it. By detecting light, the light sensor 32 can, for example, capture an image. For example, similar to the light sensor 11 in Figure 1, the light sensor 32 performs photoelectric conversion of the light incident upon it and acquires the corresponding pixel value using the CLK-DET method. Therefore, similar to the light sensor 11, the light sensor 32 can capture images with an appropriate exposure time, and can also perform BI (Broadcast Image) with a short exposure time for high-speed imaging. In addition to images for viewing, which are composed of pixel values ​​representing the light intensity of visible light, the light sensor 32 can also acquire images of other pixel values, such as distance images, which are composed of pixel values ​​representing distance. That is, the light sensor system 30 can be applied not only to capturing images for viewing, but also to distance measurement using LiDAR (light detection and ranging), etc.

[0075] The optical sensor 32 is constructed by stacking two substrates, for example, a sensor substrate 41 and a logic substrate 42. The optical sensor 32 can also be constructed from a single substrate or by stacking three or more substrates.

[0076] The sensor substrate 41 has pixels arranged in a two-dimensional plane, for example, composed of SPADs as shown in Figure 2, and is supplied with CLK-DET parameters. The sensor substrate 41 performs photoelectric conversion at the pixels and acquires pixel values ​​using the CLK-DET method. That is, according to the count period Tp and the number of periods Nmax as CLK-DET parameters, the sensor substrate 41 counts photons based on the current flowing when photons are incident on the pixels during count periods Tp equal to the number of periods Nmax in the exposure time, and acquires the photon count value obtained by counting these photons as the pixel value.

[0077] The logic board 42 is configured with a signal processing circuit (part) (not shown) that performs various signal processing on the captured image, which is composed of pixel values ​​acquired by the CLK-DET method on the sensor board 41. Furthermore, the logic board 42 is configured with an encoder 51.

[0078] The encoder 51 compresses the captured image, which is composed of pixel values ​​acquired by the CLK-DET method, based on the encoding parameters from the encoding parameter generator 31, and outputs the encoded stream obtained by this encoding to the outside of the optical sensor 32.

[0079] The decoder 33 decodes the encoded stream output by the encoder 51 based on the encoding parameters from the encoding parameter generator 31, and outputs the decoded image obtained by the decoding.

[0080] Figure 9 is a flowchart illustrating an example of the operation of the optical sensor system 30 shown in Figure 8.

[0081] In step S11, the coding parameter generator 31 generates coding parameters based on the CLK-DET parameters and supplies them to the encoder 51 and decoder 33, and the process proceeds to step S12.

[0082] In step S12, the optical sensor 32 starts capturing images using the CLK-DET method, that is, acquiring pixel values ​​according to the CLK-DET parameters from the coding parameter generator 31, and the process proceeds to step S13.

[0083] In step S13, the encoder 51 of the optical sensor 32 encodes the captured image using the CLK-DET method based on the encoding parameters from the encoding parameter generator 31, and outputs the encoded stream obtained by this encoding to the outside of the optical sensor 32.

[0084] The process then proceeds from step S13 to step S14, where the decoder 33 decodes the encoded stream output by the encoder 51 based on the encoding parameters from the encoding parameter generator 31, and outputs the decoded image obtained by the decoding.

[0085] <Example configuration of the coding parameter generator 31>

[0086] Figure 10 is a block diagram showing an example configuration of the coding parameter generator 31 shown in Figure 8.

[0087] In Figure 10, the coding parameter generator 31 includes a probability distribution calculator 61, a standard deviation calculator 62, a brightness / darkness criterion determiner 63, and a code designer 64.

[0088] The probability distribution calculator 61 is supplied with CLK-DET parameters, namely the count period Tp and the number of periods Nmax. Using the CLK-DET parameters, the probability distribution calculator 61 calculates the CLK-DET probability distribution of photons incident during the count period in the exposure time, outputs it as one of the coding parameters, and supplies it to the standard deviation calculator 62 and the code designer 64.

[0089] The standard deviation calculator 62 calculates the standard deviation (variance) of the CLK-DET probability distribution from the probability distribution calculator 61 and supplies it to the light / dark criterion determiner 63.

[0090] The brightness / darkness criterion determination unit 63 determines the brightness / darkness threshold using the standard deviation of the CLK-DET probability distribution from the standard deviation calculator 62 and outputs it as one of the coding parameters.

[0091] The code designer 64 designs entropy codes, such as Huffman codes or tANS (tabled asymmetric numeral systems) codes, based on the CLK-DET probability distribution from the standard deviation calculator 62. For example, based on the CLK-DET probability distribution, the code designer 64 can design an entropy code for the bright parts of an image (an entropy code suitable for the bright parts) and an entropy code for the dark parts of an image. The code designer 64 generates code information necessary for encoding to the designed entropy codes, such as a code table, and outputs it as one of the encoding parameters.

[0092] Figure 11 illustrates an example of calculating the CLK-DET probability distribution using the probability distribution calculator 61 shown in Figure 10.

[0093] When photons are incident on a pixel at a photon rate fp [Hz], the average number of photons incident in one counting period Tp [seconds] is Tpfp. Since the incidence of photons follows a Poisson distribution, the probability that k photons are incident in one counting period Tp is given by equation (1).

[0094] ... (1)

[0095] The probability p that one or more photons occur during a single counting period Tp is given by equation (2).

[0096] ... (2)

[0097] The exposure time has Nmax counting periods Tp. The probability P of a photon occurring during one of the Nmax counting periods Tp within the exposure time is given by P. B This follows a binomial distribution B(n, p) in which n = Nmax trials of probability p are performed, and is expressed by equation (3).

[0098] ... (3)

[0099] The exposure time can be set to include one or more counting periods Tp. If multiple counting periods T1p, T2p,... are set for the exposure time, the probability P of a photon being incident on a counting period T1p, T2p,... within the exposure time is... S This is the probability P of photons being incident during each count period, according to equation (3). B It is expressed as a product (convolution operation). For example, as shown in Figure 11, if three types of counting periods T1p, T2p, and T3p are set in the exposure time, and the number of periods for each of the three types of counting periods T1p to T3p is represented as N1max, N2max, and N3max, then the probability P that a photon is incident on a counting period T1p, T2p, or T3p within the exposure time is given by S This is expressed by equation (4).

[0100] ... (4)

[0101] When photons are incident at a photon rate fp, the CLK-DET probability distribution of photons incident during the count period in the exposure time can be calculated using equations (3) and (4).

[0102] The CLK-DET probability distribution when photons are incident at a photon rate fp is also called the CLK-DET probability distribution for the photon rate fp. Furthermore, the CLK-DET probability distribution for the photon rate fp is also called the CLK-DET probability distribution for the photon count value c, or the CLK-DET probability distribution for the pixel value c, using the photon count value c or pixel value c corresponding to that photon rate fp.

[0103] Figure 12 shows an example of the standard deviation of the CLK-DET probability distribution calculated by the standard deviation calculator 62 in Figure 10.

[0104] The standard deviation calculator 62 calculates the standard deviation of the CLK-DET probability distribution for each photon count value (each pixel value) (corresponding to each photon rate) from the probability distribution calculator 61.

[0105] Figure 12 shows an example of the standard deviation of the CLK-DET probability distribution for each photon count value. In Figure 12, the horizontal axis represents the photon count value (pixel value), and the vertical axis represents the standard deviation. Figure 12 shows that the standard deviation of the CLK-DET probability distribution changes depending on the photon count value, that is, the photon rate of photons incident on the pixel.

[0106] The standard deviation of the CLK-DET probability distribution is small for both small and large photon count values. The standard deviation of the CLK-DET probability distribution for photon count values ​​intermediate between small and large photon count values ​​will be of a certain magnitude.

[0107] Figure 13 illustrates an example of a method for determining the brightness threshold using the brightness criterion determination device 63 shown in Figure 10.

[0108] The brightness criterion determination unit 63 determines the brightness threshold using the standard deviation of the CLK-DET probability distribution for each photon count value or pixel value (corresponding to each photon rate) from the standard deviation calculator 62.

[0109] Figure 13 shows the graph of the standard deviation of the CLK-DET probability distribution in Figure 12, with the log² value of the standard deviation on the vertical axis (hereinafter also referred to as the log² graph). The vertical axis of the log² graph represents the minimum number of bits required to represent the photon count value (pixel value), which is a random variable with the same units as the standard deviation (hereinafter also referred to as the required number of bits).

[0110] By viewing the log2 graph from the smallest photon count value, the photon count value (pixel value) at which the required number of bits first becomes the reference number of bits can be determined as the brightness threshold.

[0111] The reference bit depth is the number of bits required to represent the photon count value (pixel value) that serves as the basis for allocating dark and bright areas in an image, and is predetermined. For example, the reference bit depth can be determined by simulating encoding by the encoder 51 and decoding by the decoder 33 using various values ​​as provisional reference bit depths, and evaluating the image quality of the decoded image obtained through decoding. In other words, the provisional reference bit depth that yields the (best) evaluation of the image quality of the decoded image can be determined as the final reference bit depth. In Figure 13, 4 bits are used as the reference bit depth.

[0112] Furthermore, for the sake of simplicity in the following explanation, we will define the brightness threshold as a single value that divides an image into either dark or bright areas. However, multiple values ​​can be defined as the brightness threshold. For example, two values ​​can be defined as the brightness threshold to divide an image into three parts: dark areas, bright areas that are brighter than the dark areas, and saturated areas that are brighter than the bright areas.

[0113] <Overview of Encoder 51>

[0114] The following describes the overview of the encoder 51.

[0115] The encoder 51 encodes the captured image, which consists of pixel values ​​(photon count values) acquired by the CLK-DET method on the sensor substrate 41 (Figure 8). The encoder 51 encodes the captured image using one or more of the encoding parameters, which are generated based on the CLK-DET parameters, the count period Tp and the number of periods Nmax, as needed: the CLK-DET probability distribution, the brightness threshold, and the encoding information. The encoder 51, which encodes the captured image using encoding parameters generated based on the CLK-DET parameters, can take the following first to fourth embodiments, as well as various other embodiments.

[0116] For example, the encoder 51 can take a first form in which it sequentially selects each part (region) of the captured image as a processing target, determines the brightness of the processing target, and selects an encoding method to encode the processing target based on the brightness of the processing target. For example, in the captured image, predetermined regions such as 8 pixels x 1 pixel can be selected as processing targets in the order of raster scanning.

[0117] The determination of brightness or darkness of the processing target area can be performed on the entire processing target area, or on a pixel-by-pixel basis. When the determination of brightness or darkness of the processing target area is performed on a pixel-by-pixel basis, the encoder 51 can take a second approach in which it selects an encoding method for the processing target area on a pixel-by-pixel basis based on the brightness or darkness of each pixel in the processing target area.

[0118] Furthermore, for example, in the code designer 64 (Figure 10), when designing multiple entropy codes and generating code information for each of the multiple entropy codes, the encoder 51 can take a third form in which it selects an entropy code from the multiple entropy codes to entropy encode the part to be processed, based on the part to be processed and / or the surrounding part of the part to be processed.

[0119] For example, the code designer 64 can design entropy codes for dark areas and bright areas, respectively. The encoder 51 can determine the brightness of the area to be processed and / or the surrounding area of ​​the area to be processed, and based on the determination result, can select the entropy code to be used for entropy coding of the area to be processed. That is, if the area to be processed is determined to be dark, an entropy code for the dark area can be selected, and if the area to be processed is determined to be bright, an entropy code for the bright area can be selected.

[0120] Furthermore, for example, the encoder 51 can take a fourth form in which it encodes the portion to be processed based on the CLK-DET probability distribution. For example, when the encoder 51 performs quantization in encoding the portion to be processed, it can set the quantization width for encoding the portion to be processed based on the CLK-DET probability distribution.

[0121] Figure 14 is a block diagram illustrating an overview of the configuration of an encoder 51 in a first embodiment, which determines the overall brightness of the area to be processed and selects an encoding method for the area to be processed based on the overall brightness of the area to be processed.

[0122] In Figure 14, the encoder 51 includes a representative value acquisition unit 71, a brightness / darkness determination unit 72, a brightness area encoding unit 73, a dark area encoding unit 74, and a MUX (multiplexer) 75.

[0123] The representative value acquisition unit 71 is supplied with the portion of the captured image to be processed, and / or the peripheral portion of the portion to be processed. Any region surrounding the portion to be processed can be selected as the peripheral portion. Alternatively, the peripheral portion can be selected from the region surrounding the portion to be processed, which has already been designated as the portion to be processed and encoded, for example, the upper, left, and upper left sides of the periphery of the portion to be processed.

[0124] By using the region surrounding the target area that has already been designated as the target area as the peripheral portion, encoding of the target area can be performed without waiting for pixels after the target area to be supplied to the encoder 51. That is, for example, if the captured image is supplied to the encoder 51 in raster scan order, and the peripheral portion includes pixels after the target area in raster scan order, encoding of the target area must wait until those pixels are supplied. In contrast, if the peripheral portion consists only of the region that has already been designated as the target area, all images of the peripheral portion have already been supplied to the encoder 51 by the time the target area is supplied to the encoder 51. Therefore, encoding of the target area can begin immediately after the target area is supplied to the encoder 51. In the following, the region surrounding the target area that has already been designated as the target area will be used as the peripheral portion.

[0125] The representative value acquisition unit 71 calculates a representative value for either the processing target area, the peripheral area, or both the processing target area and the peripheral area (the whole) of the captured image, and supplies this representative value to the brightness determination unit 72 as an illuminance value representing the brightness (illuminance) (luminance) of the processing target area. As the representative value for the processing target area, the average value of the pixel values ​​(photon count values) of the processing target area, the median value, or a pixel value randomly selected from the pixel values ​​of the processing target area can be used. The same applies to the representative value for the peripheral area and the representative value for both the processing target area and the peripheral area.

[0126] In the representative value acquisition unit 71, when calculating the representative value using the processing target portion, for example, the calculation of the representative value becomes possible only after the last pixel (or its pixel value) of the processing target portion supplied in raster scan order has been supplied. Therefore, the calculation of the representative value may not be completed immediately after the processing target portion is supplied. In this case, a waiting time occurs while the encoding of the processing target portion is being waited for until the calculation of the representative value is completed. This waiting time can be prevented by calculating the representative value without using the processing target portion, that is, by calculating the representative value of the peripheral portion.

[0127] The brightness determination unit 72 determines the brightness of the processing target area based on the illuminance value representing the brightness of the processing target area from the representative value acquisition unit 71.

[0128] The brightness determination unit 72 is supplied with an illuminance value representing the brightness of the processing target area from the representative value acquisition unit 71, as well as a brightness threshold, which is one of the encoding parameters from the encoding parameter generator 31 (Figure 10). Furthermore, if, for example, the light sensor system 30 (Figure 8) constitutes a camera, the brightness determination unit 72 can be supplied with camera parameters for capturing the image. Also, if, for example, the image captured by BI in the light sensor system 30 is subject to QBI signal processing, the brightness determination unit 72 can be supplied with the parameters for that QBI signal processing (QBI parameters). Camera parameters include, for example, gain and exposure (values ​​representing these values). QBI parameters include the number of images to be superimposed in QBI signal processing (number of superimposed images), etc.

[0129] The brightness determination unit 72 determines the brightness of the area to be processed based on the illuminance value of the area to be processed (a representative value for the surrounding area, etc.) and the brightness threshold. For example, the brightness determination unit 72 compares the illuminance value of the area to be processed with the brightness threshold, and determines that the area to be processed is a bright area if the illuminance value of the area to be processed is equal to (or greater than) the brightness threshold. On the other hand, the brightness determination unit 72 determines that the area to be processed is a dark area if the illuminance value of the area to be processed is less than (or less than or equal to) the brightness threshold.

[0130] The brightness determination unit 72 supplies brightness information, which is the result of determining the brightness of the processing target area, to the brightness encoding unit 73 and the darkness encoding unit 74. Here, the brightness information indicates whether the processing target area as a whole is a brightness area or a darkness area. In this case, for example, one bit can be used for the brightness information, with the brightness information set to 1 if the processing target area is a brightness area, and 0 if the processing target area is a darkness area.

[0131] The brightness determination unit 72 can determine the brightness of the processing target area using camera parameters and QBI parameters. For example, the brightness determination unit 72 can adjust the brightness threshold based on camera parameters and QBI parameters.

[0132] The bright area encoding unit 73 and the dark area encoding unit 74 are supplied with brightness information of the area to be processed from the brightness / darkness determination unit 72, as well as the area to be processed.

[0133] The bright area encoding unit 73 encodes the image using an encoding method suitable for encoding bright areas, or any encoding method other than an encoding method suitable for dark areas. If the brightness information of the area to be processed indicates that the area to be processed is a bright area, the bright area encoding unit 73 encodes the bright area to be processed and supplies the encoded stream containing the resulting encoded data to the MUX 75.

[0134] In Figure 14, the bright area coding unit 73 includes a prediction unit 81, a difference unit 82, a quantization unit 83, and a coding unit 84.

[0135] The prediction unit 81 is supplied with the portion to be processed. The prediction unit 81 generates a predicted image of the portion to be processed and supplies it to the difference unit 82 along with the portion to be processed.

[0136] The difference unit 82 calculates the difference between the processing target portion and the predicted image, and supplies it to the quantization unit 83 as a predicted residual.

[0137] The quantization unit 83 quantizes the predicted residual from the difference unit 82 and supplies the resulting quantized value to the encoding unit 84.

[0138] The encoding unit 84 performs reversible encoding, for example, entropy encoding, of the quantized values ​​from the quantization unit 83, and supplies the encoding stream of the bright portion, which includes the encoding data obtained by entropy encoding of the quantized values ​​of the bright portion, which is the processing target portion, to the MUX 75.

[0139] The dark area encoding unit 74 encodes the image using an encoding method suitable for encoding dark areas, for example, an encoding method for dark areas that suppresses image quality degradation due to encoding and decoding, particularly image quality degradation in dark areas. If the brightness information of the processing target area indicates that the processing target area is a dark area, the dark area encoding unit 74 encodes the processing target area that is a dark area and supplies the encoded stream containing the resulting encoded data to the MUX 75.

[0140] In Figure 14, the dark area coding unit 74 includes a prediction unit 91, a difference unit 92, a quantization unit 93, and a coding unit 94.

[0141] The prediction unit 91 is supplied with the portion to be processed. The prediction unit 91 generates a predicted image of the portion to be processed and supplies it to the difference unit 92 along with the portion to be processed.

[0142] The difference unit 92 calculates the difference between the processing target portion and the predicted image, and supplies it to the quantization unit 93 as a predicted residual.

[0143] The quantization unit 93 quantizes the predicted residual from the difference unit 92 and supplies the resulting quantized value to the encoding unit 94.

[0144] The encoding unit 94 performs reversible encoding, for example, entropy encoding, of the quantized values ​​from the quantization unit 93, and supplies the encoding stream of the dark region, which includes the encoding data obtained by entropy encoding of the quantized values ​​for the processing target portion, which is the dark region, to the MUX 75.

[0145] For example, in the dark area encoding unit 74, in order to suppress the deterioration of image quality in the dark areas, the quantization unit 93 may adopt an encoding method for the dark areas that uses a quantization width smaller than that of the quantization unit 83 of the bright area encoding unit 73, which employs an encoding method for the bright areas.

[0146] Furthermore, for example, in the dark area encoding unit 74, in order to suppress the deterioration of image quality in dark areas, an encoding method that does not perform quantization by the quantization unit 93 can be adopted as the encoding method for dark areas.

[0147] Furthermore, for example, in the dark area encoding unit 74, the encoding unit 94 can adopt an encoding scheme that uses an entropy code that efficiently performs entropy encoding of the dark area (performs high compression) as the encoding scheme for the dark area.

[0148] The MUX 75 multiplexes the encoded stream from the bright area encoding unit 73 and the encoded stream from the dark area encoding unit 74, and outputs the multiplexed encoded stream.

[0149] Figure 15 is a block diagram showing an overview of the configuration example of an encoder 51 in a second mode, which determines the brightness of each pixel in the area to be processed and selects the encoding method for each pixel based on the brightness of each pixel in the area to be processed.

[0150] In the figures, parts corresponding to those in Figure 14 are denoted by the same reference numerals, and their explanations will be omitted as appropriate below.

[0151] In Figure 15, the encoder 51 includes a bright area encoding unit 73, a MUX 75, a brightness / darkness determination unit 111, and a dark area encoding unit 112. Therefore, in Figure 15, the encoder 51 is the same as in Figure 14 in that it includes a bright area encoding unit 73 and a MUX 75. However, the encoder 51 in Figure 15 differs from the one in Figure 14 in that a brightness / darkness determination unit 111 is provided instead of a representative value acquisition unit 71 and a brightness / darkness determination unit 72, and a dark area encoding unit 112 is provided instead of a dark area encoding unit 74.

[0152] The brightness determination unit 111 is supplied with the portion of the captured image to be processed. The brightness determination unit 111 is also supplied with a brightness threshold, which is one of the encoded parameters from the encoded parameter generator 31 (Figure 10). Furthermore, the brightness determination unit 111 can be supplied with camera parameters and QBI parameters.

[0153] The brightness determination unit 111 determines the brightness of each pixel in the processing target area based on the processing target area and the brightness threshold. For example, the brightness determination unit 111 compares the pixel value of each pixel in the processing target area with the brightness threshold, and determines that the corresponding pixel is a bright area if the pixel value is equal to (or greater than) the brightness threshold. On the other hand, the brightness determination unit 111 determines that the corresponding pixel is a dark area if the pixel value is less than (or less than) the brightness threshold.

[0154] The brightness determination unit 111 supplies brightness information, which is the result of determining the brightness of the processing target area, to the brightness encoding unit 73 and the darkness encoding unit 112. Here, the brightness information indicates whether each pixel of the processing target area is a brightness area or a darkness area. In this case, for example, the brightness information can be a bit sequence equal to the number of pixels constituting the processing target area, with bits corresponding to brightness pixels in the processing target area set to 1 and bits corresponding to darkness pixels set to 0. Such brightness information becomes a bit pattern representing the brightness of each pixel in the processing target area.

[0155] The brightness determination unit 111, similar to the brightness determination unit 72 in Figure 14, can determine the brightness of each pixel in the processing area using camera parameters and QBI parameters. For example, the brightness determination unit 111 can adjust the brightness threshold based on camera parameters and QBI parameters.

[0156] The bright area encoding unit 73 and the dark area encoding unit 112 are supplied with brightness information of the area to be processed from the brightness / darkness determination unit 111, as well as the area to be processed.

[0157] The bright area encoding unit 73 encodes at least the bright pixels of the processing target area based on the brightness information, in the same manner as in Figure 14, and supplies the encoded stream containing the resulting encoded data to the MUX 75.

[0158] The dark area encoding unit 112 encodes the image using an encoding method for dark areas. Based on the brightness information, the dark area encoding unit 112 encodes the pixels in the dark areas of the processing target using an encoding method for dark areas that suppresses the degradation of image quality in dark areas, and supplies the encoded stream containing the resulting encoded data to the MUX 75.

[0159] In the dark area encoding unit 112, in order to suppress the degradation of image quality in dark areas, an encoding method is adopted for dark areas that does not calculate predicted residuals or perform quantization, but instead reversibly encodes (entropy encodes) the pixel values ​​as they are. Therefore, the dark area encoding unit 112 does not have the prediction unit 91, difference unit 92, and quantization unit 93 shown in Figure 14, but has an encoding unit 94.

[0160] The encoding unit 94 entropy encodes the pixels (or their pixel values) in the dark areas of the pixels to be processed, and supplies the encoded stream of the dark areas, which includes the encoded data obtained by the entropy encoding, to the MUX 75.

[0161] In addition, the dark area encoding unit 112 can also employ an encoding method for the dark area, similar to the dark area encoding unit 74, which involves calculating the predicted residual, quantizing it, and then entropy encoding the quantized value.

[0162] Figure 16 is a block diagram illustrating an overview of the configuration example of a third embodiment of an encoder 51 that selects an entropy code to entropically encode the portion to be processed from a plurality of entropy codes based on the portion to be processed and / or the surrounding portion of the portion to be processed.

[0163] In the figures, parts corresponding to those in Figure 14 or Figure 15 are denoted by the same reference numerals, and their explanations will be omitted as appropriate below.

[0164] In Figure 16, the encoder 51 includes a representative value acquisition unit 71, a brightness / darkness determination unit 72, a brightness area encoding unit 73, a MUX 75, and a dark area encoding unit 112. Therefore, in Figure 16, the encoder 51 is the same as in Figure 14 in that it includes the representative value acquisition unit 71 to the brightness area encoding unit 73 and the MUX 75. However, the encoder 51 in Figure 16 differs from the one in Figure 14 in that the dark area encoding unit 112 of Figure 15 is provided instead of the dark area encoding unit 74.

[0165] In Figure 16, the brightness determination unit 72 is not supplied with camera parameters or QBI parameters. However, in Figure 16, as in Figure 14, the brightness determination of the processing target area can also be performed using camera parameters and QBI parameters.

[0166] The bright area encoding unit 73 and the dark area encoding unit 112 are supplied with brightness information of the processing target area from the brightness / darkness determination unit 101, as well as the processing target area itself.

[0167] As explained in Figure 14, if the brightness information of the processing target area supplied from the brightness determination unit 72 indicates that the processing target area is a bright area, the brightness encoding unit 73 encodes the bright processing target area using a brightness encoding scheme, and supplies the encoded stream containing the resulting encoded data to the MUX 75.

[0168] However, in Figure 16, the encoding unit 84 of the bright area encoding unit 73 is supplied with code information, which is one of the encoding parameters from the encoding parameter generator 31 (Figure 10). Based on the code information, the encoding unit 84 entropy encodes the portion to be processed. The code information includes multiple code tables, for example, a code table for entropy codes for bright areas (code table for bright areas) and a code table for entropy codes for dark areas (code table for dark areas). The encoding unit 84 selects the code table for bright areas and entropy encodes the quantized values ​​for the bright portion to be processed from the quantization unit 83 based on that code table for bright areas. Therefore, in the encoding unit 84, from the multiple entropy codes, including the entropy codes for bright areas and the entropy codes for dark areas, the entropy code for bright areas is selected as the entropy code to entropy encode the bright portion to be processed, and the bright portion to be processed is encoded with the entropy code for bright areas.

[0169] The encoding unit 84 supplies the MUX 75 with an encoded stream of the bright portion, which includes encoded data obtained by entropy encoding of the quantization value of the bright portion, the part to be processed.

[0170] If the brightness information of the processing target area supplied from the brightness determination unit 72 indicates that the processing target area is a dark area, the dark area encoding unit 112 encodes the processing target area that is a dark area using a dark area encoding method, and supplies the encoded stream containing the resulting encoded data to the MUX 75.

[0171] As explained in Figure 15, the dark area coding unit 112 employs an encoding method for the dark area that entropy encodes the pixel values ​​as they are, without calculating the predicted residual or performing quantization. Therefore, the dark area coding unit 112 does not have the prediction unit 91, difference unit 92, and quantization unit 93 shown in Figure 14, but has an encoding unit 94.

[0172] In Figure 16, the encoding unit 94 of the dark area encoding unit 112 is supplied with code information, which is one of the encoding parameters from the encoding parameter generator 31 (Figure 10). Based on the code information, the encoding unit 94 entropically encodes the area to be processed. For example, as described above, the code information includes a code table for the bright area and a code table for the dark area. The encoding unit 94 selects the code table for the dark area and entropically encodes the dark area to be processed based on that code table. Therefore, the encoding unit 94 selects the entropy code for the dark area from among multiple entropy codes, including the entropy code for the bright area and the entropy code for the dark area, and the dark area to be processed is encoded with the entropy code for the dark area.

[0173] Figure 17 is a block diagram showing an overview of an example configuration of a fourth type of encoder 51 that encodes the portion to be processed based on the CLK-DET probability distribution.

[0174] In the figures, parts corresponding to those in Figure 16 are denoted by the same reference numerals, and their explanations will be omitted as appropriate below.

[0175] In Figure 17, the encoder 51 includes a representative value acquisition unit 71, a brightness / darkness determination unit 72, a brightness area encoding unit 73, a MUX 75, and a dark area encoding unit 112. Therefore, in Figure 17, the encoder 51 is configured in the same way as in Figure 16. However, in Figure 17, the quantization unit 83 of the brightness area encoding unit 73 is supplied with a CLK-DET probability distribution, which is one of the encoding parameters from the encoding parameter generator 31 (Figure 10).

[0176] In Figure 17, the quantization unit 83 sets the final quantization width to be used for quantization of the part to be processed based on the CLK-DET probability distribution, and then performs quantization.

[0177] The schematic configuration of the decoder 33, which decodes the encoded stream obtained by the encoder 51 of the first to fourth embodiments, is such that the processing performed in the bright area encoding unit 73, the dark area encoding units 74 and 112, and the MUX 75 are performed in reverse order, so a detailed explanation is omitted.

[0178] <First configuration example of encoder 51>

[0179] Figure 18 is a block diagram showing a first configuration example of the encoder 51 in Figure 8.

[0180] In Figure 18, the encoder 51 includes a representative value acquisition unit 121, a brightness / darkness determination unit 122, and an encoder core 123.

[0181] The representative value acquisition unit 121 is supplied with the processing target portion of the captured image, and / or the peripheral portion of the processing target portion. Similar to the representative value acquisition unit 71 in Figure 14, the representative value acquisition unit 121 calculates a representative value for either the processing target portion, the peripheral portion, or both the processing target portion and the peripheral portion of the captured image, and supplies this representative value to the brightness determination unit 122 as an illuminance value representing the brightness (illuminance) of the processing target portion.

[0182] The brightness determination unit 122, similar to the brightness determination unit 72 in Figure 14, determines the brightness of the processing target area based on the illuminance value representing the brightness of the processing target area from the representative value acquisition unit 121.

[0183] The brightness determination unit 122 is supplied with an illuminance value representing the brightness of the processing target area from the representative value acquisition unit 121, as well as a brightness threshold, which is one of the coding parameters, from the coding parameter generator 31 (Figure 10). Furthermore, for example, if the light sensor system 30 (Figure 8) constitutes a camera, the brightness determination unit 122 can be supplied with camera parameters for capturing the image. Also, for example, if the image captured by the light sensor system 30 using BI is subject to QBI signal processing, the parameters for that QBI signal processing (QBI parameters) can be supplied.

[0184] The brightness determination unit 122, similar to the brightness determination unit 72 in Figure 14, determines the brightness of the area to be processed based on the illuminance value of the area to be processed (a representative value for the surrounding area, etc.) and a brightness threshold. For example, the brightness determination unit 122 compares the illuminance value of the area to be processed with the brightness threshold, and determines that the area to be processed is a bright area if the illuminance value of the area to be processed is equal to or greater than the brightness threshold. On the other hand, the brightness determination unit 122 determines that the area to be processed is a dark area if the illuminance value of the area to be processed is less than the brightness threshold.

[0185] The brightness determination unit 122 generates brightness information, which is the result of determining the brightness of the processing target area, and supplies it to the encoder core 123. The brightness information generated by the brightness determination unit 122 indicates whether the processing target area as a whole is a bright area or a dark area. In this case, as in the case of Figure 14, for example, one bit can be used for the brightness information, with the brightness information set to 1 if the processing target area is a bright area, and 0 if the processing target area is a dark area.

[0186] The brightness determination unit 122 can determine the brightness of the processing target area using camera parameters and QBI parameters. For example, the brightness determination unit 122 can adjust the brightness threshold based on camera parameters and QBI parameters.

[0187] The encoder core 123 is supplied with brightness / darkness information from the brightness / darkness determination unit 122, as well as a processing target portion whose brightness / darkness is represented by that brightness / darkness information. Furthermore, the encoder core 123 is supplied with coding parameters, namely the CLK-DET probability distribution and / or coding information, from the coding parameter generator 31 (Figure 10) as needed.

[0188] The encoder core 123 encodes the portion to be processed based on the brightness threshold, that is, based on the brightness information generated by the brightness determination unit 122 using the brightness threshold, and generates and outputs an encoded stream containing the encoded data of the portion to be processed. In the encoder core 123, encoding of the portion to be processed can also be performed based on the CLK-DET probability distribution and / or the encoding information, if necessary.

[0189] Figure 19 is a block diagram showing an example configuration of the light / dark determination unit 122 in Figure 18.

[0190] In Figure 19, the brightness determination unit 122 has a comparison unit 131.

[0191] The comparison unit 131 is supplied with an illuminance value representing the brightness of the area to be processed from the representative value acquisition unit 121, and a brightness threshold value from the encoding parameter generator 31 (Figure 10). The comparison unit 131 compares the illuminance value of the area to be processed with the brightness threshold value, and determines that the area to be processed is a bright area if the illuminance value of the area to be processed is equal to (or greater than) the brightness threshold value. On the other hand, the comparison unit 131 determines that the area to be processed is a dark area if the illuminance value of the area to be processed is less than (or less than or equal to) the brightness threshold value. The comparison unit 131 generates brightness information, which is the result of the brightness determination of the area to be processed, and supplies it to the encoding core 123.

[0192] Figure 20 is a block diagram showing another example of the configuration of the light / dark determination unit 122 in Figure 18.

[0193] In the figures, parts corresponding to those in Figure 19 are denoted by the same reference numerals, and their explanations will be omitted as appropriate below.

[0194] In Figure 20, the brightness determination unit 122 has a comparison unit 131 and a threshold adjustment unit 141. Therefore, the brightness determination unit 122 in Figure 20 is the same as in Figure 19 in that it has a comparison unit 131, but differs from the case in Figure 19 in that a threshold adjustment unit 141 is newly provided.

[0195] The threshold adjustment unit 141 is supplied with a brightness threshold from the coding parameter generator 31 (Figure 10), along with camera parameters and / or QBI parameters. Based on the camera parameters and / or QBI parameters, the threshold adjustment unit 141 adjusts the brightness threshold and supplies the adjusted threshold to the comparison unit 131.

[0196] Therefore, in Figure 20, the comparison unit 131 compares the illuminance value of the processing target area with the brightness threshold adjusted based on the camera parameters and / or QBI parameters.

[0197] For example, if camera parameters such as gain and exposure are high, the captured image is assumed to be bright overall. In this case, in order to relatively determine the brightness and darkness of the area to be processed within such a brightly captured image, the brightness and darkness threshold in the threshold adjustment unit 141 is adjusted to a high value according to the gain and exposure. On the other hand, if the gain and exposure are low, the captured image is assumed to be dark overall. In this case, in order to relatively determine the brightness and darkness of the area to be processed within such a darkly captured image, the brightness and darkness threshold in the threshold adjustment unit 141 is adjusted to a low value according to the gain and exposure.

[0198] Figure 21 illustrates an example of adjusting the brightness threshold based on the QBI parameters in the threshold adjustment unit 141 of Figure 20.

[0199] Figure 21 shows the relationship between the number of superimposed images, which is a QBI parameter, and the camera parameters, gain and exposure.

[0200] In QBI signal processing, the number of superimposed images needs to be greater the lower the exposure of the captured image (darker image), and less the higher the exposure (brighter image). Therefore, there is a negative correlation between the number of superimposed images and the gain and exposure, as shown in Figure 21.

[0201] Therefore, when the number of superimposed images is small, the threshold adjustment unit 141 can adjust the brightness threshold to a high value in accordance with the number (or lack thereof) of superimposed images, similar to when the gain and exposure are high. On the other hand, when the number of superimposed images is large, the threshold adjustment unit 141 can adjust the brightness threshold to a low value in accordance with the number (or lack thereof) of superimposed images, similar to when the gain and exposure are low.

[0202] When adjusting the brightness threshold based on multiple elements such as the number of overlaps, gain, and exposure, the brightness threshold can be adjusted by a value based on the adjustment amount for each individual element, such as an average value. For example, when adjusting the brightness threshold based on all elements of the number of overlaps, gain, and exposure, the brightness threshold can be adjusted by the average value of the adjustment amounts calculated based on each of the elements: the number of overlaps, gain, and exposure.

[0203] Figure 22 is a block diagram showing an example configuration of the encoder core 123 in Figure 18.

[0204] In Figure 22, the encoder core 123 includes a prediction method determination unit 151, a prediction unit 152, a calculation unit 153, a quantization unit 154, an inverse quantization unit 155, a calculation unit 156, a reference buffer 157, a reversible encoding unit 158, and a stream generation unit 159.

[0205] The prediction method determination unit 151 is supplied with the processing target portion, as well as a reference image (including necessary portions such as the peripheral portion of the processing target portion) stored in the reference buffer 157, which is an already encoded and locally decoded image. Based on the processing target portion and the reference image, the prediction method determination unit 151 determines the prediction mode when performing prediction during encoding of the processing target portion, and the quantization width when performing quantization. The prediction method determination unit 151 supplies the prediction mode to the prediction unit 152 and the stream generation unit 159, and supplies the quantization width to the quantization unit 154, the inverse quantization unit 155, and the stream generation unit 159. Note that, to avoid making the diagram complicated, the illustration of the connection lines supplying the quantization width (to the quantization unit 154, etc.) has been omitted. The same applies to the diagram described later.

[0206] The prediction unit 152 is supplied with a prediction mode from the prediction method determination unit 151, as well as a reference image from the reference buffer 157. Using the reference image from the reference buffer 157, the prediction unit 152 predicts a predicted image of the processing target portion using a prediction method according to the prediction mode from the prediction method determination unit 151, and supplies it to the calculation units 153 and 156.

[0207] The calculation unit 153 is supplied with a predicted image from the prediction unit 152, as well as the portion to be processed. The calculation unit 153 generates a predicted residual by calculating the difference between the portion to be processed and the predicted image, and supplies it to the quantization unit 154.

[0208] The quantization unit 154 quantizes the predicted residual from the calculation unit 153 using the quantization width determined by the prediction method determination unit 151, and supplies the resulting quantized value to the inverse quantization unit 155 and the reversible coding unit 158.

[0209] The inverse quantization unit 155 inversely quantizes the quantized value of the quantization unit 154 with the same quantization width as the quantization width in the quantization unit 154, in this case, the quantization width determined by the prediction method determination unit 151, and supplies the resulting prediction residual as an inverse quantized value to the calculation unit 156.

[0210] The calculation unit 156 (locally) decodes the processing target portion by adding the predicted residual from the inverse quantization unit 155 and the predicted image from the prediction unit 152, and supplies the decoded image obtained by this decoding to the reference buffer 157.

[0211] The reference buffer 157 stores the decoded image from the calculation unit 156 as a reference image to be used in the generation of the prediction image. The necessary reference images stored in the reference buffer 157 are supplied to the prediction method determination unit 151 and the prediction unit 152.

[0212] The reversible coding unit 158 ​​is supplied with quantized values ​​for the processing target portion from the quantization unit 154, as well as brightness / darkness information for the processing target portion from the brightness / darkness determination unit 122 (Figure 18). In addition, the reversible coding unit 158 ​​is supplied with the CLK-DET probability distribution and / or coding information from the coding parameter generator 31 (Figure 10) as needed.

[0213] The reversible coding unit 158 ​​performs reversible coding of the quantized values ​​of the portion to be processed from the quantization unit 154, for example, entropy coding, and supplies the coded data, which is the result of the entropy coding, to the stream generation unit 159.

[0214] In the reversible coding unit 158, entropy coding can be performed based on brightness information (the brightness of the processing target area represented by it). That is, the reversible coding unit 158 ​​can select an appropriate entropy code as the coding method for the processing target area based on the brightness information. For example, the reversible coding unit 158 ​​can select an appropriate code table for entropy coding of the processing target area from multiple code tables categorized by illuminance (brightness) of the image, based on the brightness information.

[0215] For example, the reversible coding unit 158 ​​can store pre-generated code tables for bright areas and code tables for dark areas. In this case, when the brightness information indicates that the area to be processed is a bright area, the reversible coding unit 158 ​​selects the code table for bright areas and can entropically encode the area to be processed (more precisely, the quantized value of the area to be processed) based on that code table. Also, when the brightness information indicates that the area to be processed is a dark area, the reversible coding unit 158 ​​selects the code table for dark areas and can entropically encode the area to be processed based on that code table.

[0216] For example, the reversible coding unit 158 ​​can be supplied with coding information from the coding parameter generator 31 (Figure 10), which includes a code table for bright areas and a code table for dark areas, as coding parameters generated based on the CLK-DET parameters. In this case, the reversible coding unit 158 ​​can entropically encode the processing target portion (more precisely, the quantized value of the processing target portion) based on the coding information. For example, if the brightness / darkness information indicates that the processing target portion is a bright area, the reversible coding unit 158 ​​can select the code table for bright areas from the coding information and entropically encode the processing target portion using that code table. Also, if the brightness / darkness information indicates that the processing target portion is a dark area, the reversible coding unit 158 ​​can select the code table for dark areas from the coding information and entropically encode the processing target portion using that code table.

[0217] Here, in the coding parameter generator 31 (Figure 10), the probability distribution calculator 61 can calculate, for example, the CLK-DET probability distribution for each possible value of the photon count value (pixel value), i.e., each value (integer value) within the range from the minimum value of the photon count value (corresponding to the photon rate) to the maximum value of the photon count value. The minimum and maximum values ​​of the photon count value are, for example, 0 and the total number of count periods within the exposure time, respectively.

[0218] In the coding parameter generator 31, the code designer 64 can design an entropy code based on, for example, the CLK-DET probability distribution for each possible value of the photon count value (pixel value). The code designer 64 can then generate multiple code tables in which each entropy code is registered as an entropy code for the dark areas. Furthermore, the code designer 64 can design an entropy code based on, for example, the CLK-DET probability distribution for a predetermined value among the possible values ​​of the photon count value (pixel value) that is smaller than the brightness threshold. The code designer 64 can then generate a code table for the dark areas in which this entropy code is registered as an entropy code for the dark areas. In addition, the code designer 64 can design an entropy code based on, for example, the CLK-DET probability distribution for a predetermined value among the possible values ​​of the photon count value (pixel value) that is larger than the brightness threshold. The code designer 64 can then generate a code table for the bright areas in which this entropy code is registered as an entropy code for the bright areas.

[0219] Among the possible values ​​for the photon count value (pixel value), a predetermined value smaller than the brightness threshold can be, for example, a value that divides the space between the minimum photon count value of 0 and the brightness threshold equally. Among the possible values ​​for the photon count value (pixel value), a predetermined value larger than the brightness threshold can be, for example, a value that divides the space between the brightness threshold and the maximum photon count value equally.

[0220] In the coding parameter generator 31, the probability distribution calculator 61 can calculate a CLK-DET probability distribution for each of the N-1 values ​​that divide the range between the minimum and maximum values ​​of the photon count value (pixel value) into N (>=3) equal parts, rather than all possible values ​​of the photon count value (pixel value). In this case, the code designer 64 can design N-1 (types) of entropy codes based on the CLK-DET probability distribution for each of the N-1 values ​​that are a subset of the possible values ​​of the photon count value (pixel value). Furthermore, the code designer 64 can design an entropy code for dark areas based on the CLK-DET probability distribution for the value among the N-1 values ​​that are a subset of the possible values ​​of the photon count value (pixel value) that is closest to a predetermined value smaller than the brightness threshold. Similarly, the code designer 64 can design an entropy code for bright areas based on the CLK-DET probability distribution for the value closest to a predetermined value greater than the brightness threshold, out of N-1 possible values ​​for the photon count value (pixel value).

[0221] For example, the reversible coding unit 158 ​​can be supplied with the CLK-DET probability distribution calculated as described above from the coding parameter generator 31. In this case, the reversible coding unit 158 ​​can perform entropy coding of the processing target portion (more precisely, the quantized value of the processing target portion) using an entropy code (hereinafter also called a direct entropy code), such as a rANS (range asymmetric numeral systems) code or a sANS (static asymmetric numeral systems) code, which can be used to perform direct entropy coding without using a code table, based on the CLK-DET probability distribution (and brightness / darkness information). For example, if the brightness / darkness information indicates that the processing target portion is a bright area, the reversible coding unit 158 ​​can entropy code the processing target portion with a direct entropy code using a calculation formula that uses the CLK-DET probability distribution from the coding parameter generator 31 for the value closest to a predetermined value greater than the brightness / darkness threshold. Furthermore, if the brightness information indicates that the area to be processed is a dark area, the reversible coding unit 158 ​​can directly entropy code the area to be processed using an entropy coding method that uses a calculation formula based on the CLK-DET probability distribution from the coding parameter generator 31, which corresponds to the value closest to a predetermined value smaller than the brightness threshold.

[0222] The stream generation unit 159 is supplied with a prediction mode and quantization width from the prediction method determination unit 151, as well as encoded data from the reversible encoding unit 158, and brightness / darkness information from the brightness / darkness determination unit 122 (Figure 18). The stream generation unit 159 generates and outputs an encoded stream in a predetermined format that includes the prediction mode and quantization width from the prediction method determination unit 151, the brightness / darkness information from the brightness / darkness determination unit 122, and the encoded data from the reversible encoding unit 158.

[0223] Figure 23 is a block diagram showing an example configuration of the reversible encoding unit 158 ​​in Figure 22.

[0224] In Figure 23, the reversible coding unit 158 ​​includes a table selection unit 171 and an entropy coding unit 172.

[0225] The table selection unit 171 is supplied with brightness information from the brightness determination unit 122 (Figure 18), as well as encoding information from the encoding parameter generator 31 (Figure 10). The encoding information includes, for example, an encoding table for bright areas and an encoding table for dark areas.

[0226] The table selection unit 171, based on the brightness / darkness information, selects a code table from among multiple code tables included in the code information, such as a code table for bright areas and a code table for dark areas, that is appropriate for entropy coding the processing target portion (more precisely, the quantized value for the processing target portion) that the brightness / darkness information represents. This code table (a code table that performs efficient (data-reducing) entropy coding) is selected and supplied to the entropy coding unit 172 as a selection table. For example, if the brightness / darkness information represents a bright area (the processing target portion is a bright area), the table selection unit 171 selects a code table for bright areas, and if the brightness / darkness information represents a dark area (the processing target portion is a dark area), it selects a code table for dark areas.

[0227] The entropy coding unit 172 is supplied with a selection table from the table selection unit 171, as well as quantized values ​​for the processing target portion from the quantization unit 154. The entropy coding unit 172 entropy codes the quantized values ​​for the processing target portion based on the selection table, and supplies the encoded data obtained by this entropy coding to the stream generation unit 159 (Figure 22).

[0228] Figure 24 illustrates the generation of a code table for bright areas and a code table for dark areas, which are included in the code information supplied to the table selection unit 171 in Figure 23.

[0229] In the coding parameter generator 31 (Figure 10), the code designer 64 designs an entropy code based on a CLK-DET probability distribution for a predetermined value among the possible values ​​of the photon count value (pixel value) that is smaller than the brightness threshold, and generates a code table for the dark areas by registering that entropy code as the entropy code for the dark areas. Furthermore, the code designer 64 designs an entropy code based on a CLK-DET probability distribution for a predetermined value among the possible values ​​of the photon count value (pixel value) that is larger than the brightness threshold, and generates a code table for the bright areas by registering that entropy code as the entropy code for the bright areas.

[0230] In Figure 24, 10 is adopted as a predetermined value smaller than the brightness threshold, and a code table for dark areas is generated in which the photon count value (pixel value) and the entropy code are associated based on the CLK-DET probability distribution for the photon count value (pixel value) = 10. Furthermore, 100 is adopted as a predetermined value larger than the brightness threshold, and a code table for bright areas is generated in which the photon count value (pixel value) and the entropy code are associated based on the CLK-DET probability distribution for the photon count value (pixel value) = 100.

[0231] The table selection unit 171 selects a code table for bright areas as the selection table for processing targets where the brightness information indicates a bright area, and selects a code table for dark areas as the selection table for processing targets where the brightness information indicates a dark area.

[0232] Note that in Figure 24, for the sake of simplicity, the photon count value (pixel value) itself is used as the random variable of the CLK-DET probability distribution (the horizontal axis of the CLK-DET probability distribution in Figure 24), and therefore the random variable of the CLK-DET probability distribution is a value of 0 or greater. When entropy coding the predicted residual (quantized value) of the processing target part, the random variable of the CLK-DET probability distribution becomes the predicted residual, and can take not only a value of 0 or greater, but also a negative value.

[0233] Figure 25 is a block diagram showing another configuration example of the reversible encoding unit 158 ​​in Figure 22.

[0234] In Figure 25, the reversible coding unit 158 ​​has an entropy coding unit 181.

[0235] The entropy coding unit 181 is supplied with brightness information from the brightness determination unit 122 (Figure 18), as well as a CLK-DET probability distribution from the coding parameter generator 31 (Figure 10). Furthermore, the entropy coding unit 181 is supplied with quantized values ​​for the portion to be processed from the quantization unit 154.

[0236] The entropy coding unit 181 entropy codes the portion to be processed (more precisely, the quantized value of the portion to be processed) using a direct entropy code such as an rANS code or a sANS code, based on the CLK-DET probability distribution and the brightness / darkness information. In other words, the entropy coding unit 181 selects a CLK-DET probability distribution based on the brightness / darkness information, and then entropy codes the portion to be processed using a direct entropy code such as an rANS code or a sANS code, based on the selected CLK-DET probability distribution.

[0237] For example, if the brightness information indicates that the area to be processed is a bright area, the entropy coding unit 181 selects the CLK-DET probability distribution for bright areas from the CLK-DET probability distributions from the coding parameter generator 31, and entropically codes the area to be processed directly into an entropy code using a calculation formula that utilizes that CLK-DET probability distribution for bright areas. Also, if the brightness information indicates that the area to be processed is a dark area, the entropy coding unit 181 selects the CLK-DET probability distribution for dark areas from the CLK-DET probability distributions from the coding parameter generator 31, and entropically codes the area to be processed directly into an entropy code using a calculation formula that utilizes that CLK-DET probability distribution for dark areas.

[0238] For bright areas, a CLK-DET probability distribution can be adopted for a predetermined large photon count value (pixel value) that represents the bright areas. This predetermined large photon count value could be, for example, a value greater than the brightness threshold, or a relatively large value among the possible photon count values ​​regardless of the brightness threshold. Similarly, for dark areas, a CLK-DET probability distribution can be adopted for a predetermined small photon count value (pixel value) that represents the dark areas. In this case, this predetermined small photon count value could be, for example, a value smaller than the brightness threshold, or a relatively small value among the possible photon count values ​​regardless of the brightness threshold.

[0239] Figure 26 illustrates an example of an encoded stream generated by the stream generation unit 159 in Figure 22.

[0240] Figure 26 shows an example of the format of the portion of the encoded stream generated by the stream generation unit 159 that corresponds to the portion to be processed (hereinafter also referred to as the unit). The format in Figure 26 is also referred to as the first format.

[0241] A unit consists of a header and a payload that follows that header.

[0242] The header includes brightness / contrast information of the area to be processed, the prediction mode, and the quantization width. In Figure 26, as explained in Figure 14, one bit is used for brightness / contrast information. If the area to be processed is bright, the brightness / contrast information is set to 1, and if the area to be processed is dark, the brightness / contrast information is set to 0.

[0243] The payload contains encoded data for the part to be processed.

[0244] <First configuration example of decoder 33>

[0245] Figure 27 is a block diagram showing a first configuration example of the decoder 33 in Figure 8.

[0246] In Figure 27, the decoder 33 includes a stream expansion unit 201, a reversible decoding unit 202, an inverse quantization unit 203, a prediction unit 204, an arithmetic unit 205, and a reference buffer 206. The decoder 33 decodes the encoded stream obtained by the encoding of the encoder 51 in Figure 22 using a decoding method corresponding to the encoding method of the encoder 51.

[0247] The stream expansion unit 201 is supplied with the encoded stream obtained by the encoding of the encoder 51 in Figure 19. The stream expansion unit 201 acquires (receives) the encoded stream and expands it to extract the brightness / darkness information of the processing target portion, the prediction mode, the quantization width, and the encoded data from the encoded stream. The stream expansion unit 201 supplies the encoded data and brightness / darkness information to the reversible decoding unit 202, and supplies the quantization width to the inverse quantization unit 203. Furthermore, the stream expansion unit 201 supplies the prediction mode to the prediction unit 204.

[0248] The reversible decoding unit 202 performs the same processing (reversible decoding back to the original) as the reversible encoding unit 158 ​​of the encoder core 123 (Figure 22). The inverse quantization unit 203, prediction unit 204, arithmetic unit 205, and reference buffer 206 perform the same processing as the inverse quantization unit 155, prediction unit 152, arithmetic unit 156, and reference buffer 157 of the encoder core 123, respectively.

[0249] The reversible decoding unit 202 is supplied with encoded data and brightness / darkness information from the stream expansion unit 201, as well as the CLK-DET probability distribution and / or coded information from the coding parameter generator 31 (Figure 10) as needed.

[0250] The reversible decoding unit 202 performs reversible decoding of the encoded data from the stream expansion unit 201, for example, entropy decoding, and supplies the quantization value for the portion to be processed, which is the result of the entropy decoding, to the inverse quantization unit 203.

[0251] The reversible decoding unit 202 performs entropy decoding corresponding to the entropy coding of the reversible coding unit 158 ​​(Figure 22), and restores the encoded data, which is an entropy code obtained by the entropy coding of the reversible coding unit 158, to its original form.

[0252] Therefore, for example, if the reversible coding unit 158 ​​stores pre-generated code tables for bright areas and code tables for dark areas, and selects either the code table for bright areas or the code table for dark areas based on the brightness / darkness information, and performs entropy coding based on the selected code table, then the reversible decoding unit 202 also stores the same code tables for bright areas and code tables for dark areas as the reversible coding unit 158, and selects either the code table for bright areas or the code table for dark areas based on the brightness / darkness information, and performs entropy decoding based on the selected code table.

[0253] Furthermore, for example, if the reversible coding unit 158 ​​selects either a code table for bright areas or a code table for dark areas from the code information supplied from the coding parameter generator 31 (Figure 10) based on the brightness / darkness information, and performs entropy coding based on the selected code table, the reversible decoding unit 202 also selects either a code table for bright areas or a code table for dark areas from the code information supplied from the coding parameter generator 31 based on the brightness / darkness information, and performs entropy decoding based on the selected code table.

[0254] Furthermore, for example, if the reversible coding unit 158 ​​selects a CLK-DET probability distribution based on the brightness information and performs entropy coding to a direct entropy code such as a rANS code or a sANS code using a calculation formula based on the selected CLK-DET probability distribution, the reversible decoding unit 202 also selects a CLK-DET probability distribution to be used in the calculation formula for entropy decoding from the CLK-DET probability distribution supplied from the coding parameter generator 31 based on the brightness information, and performs entropy decoding using a calculation formula based on the selected CLK-DET probability distribution.

[0255] The inverse quantization unit 203 inversely quantizes the quantized value from the reversible decoding unit 202 using the quantization width from the stream expansion unit 201, and supplies the resulting predicted residual, which is the inverse quantized value, to the calculation unit 205.

[0256] The prediction unit 204 is supplied with a prediction mode from the stream expansion unit 201, as well as a reference image from the reference buffer 206. Using the reference image from the reference buffer 206, the prediction unit 204 predicts a predicted image of the processing target portion using a prediction method according to the prediction mode from the stream expansion unit 201, and supplies it to the calculation unit 205.

[0257] The calculation unit 205 decodes the processing target portion by adding the predicted residual from the inverse quantization unit 203 and the predicted image from the prediction unit 204, and supplies the decoded image obtained by the decoding to the reference buffer 206 and also outputs it externally.

[0258] The reference buffer 206 stores the decoded image from the arithmetic unit 205 as a reference image to be used in the generation of the prediction image. The necessary reference images stored in the reference buffer 206 are supplied to the prediction unit 204.

[0259] Figure 28 is a block diagram showing an example configuration of the reversible decoding unit 202 in Figure 27.

[0260] In Figure 28, the reversible decoding unit 202 has a table selection unit 211 and an entropy decoding unit 212, and performs entropy decoding corresponding to the entropy coding performed by the reversible coding unit 158 ​​in Figure 23.

[0261] The table selection unit 211 is supplied with brightness / darkness information from the stream expansion unit 201 (Figure 27), as well as encoding information from the encoding parameter generator 31 (Figure 10). The encoding information includes, for example, an encoding table for bright areas and an encoding table for dark areas.

[0262] Based on the brightness / darkness information, the table selection unit 211 selects a code table from a plurality of code tables included in the code information, such as a code table for bright areas and a code table for dark areas, to be used for entropy decoding of the encoded data of the processing target portion where the brightness / darkness information represents brightness, in the same manner as the table selection unit 171 in Figure 23, and supplies it to the entropy decoding unit 212 as a selected table.

[0263] The entropy decoding unit 212 is supplied with a selection table from the table selection unit 211, as well as encoded data from the stream expansion unit 201 (Figure 27). The entropy decoding unit 212 entropy decodes the encoded data based on the selection table, and supplies the quantization value for the processing target portion obtained by the entropy decoding to the inverse quantization unit 203 (Figure 27).

[0264] Figure 29 is a block diagram showing another configuration example of the reversible decoding unit 202 in Figure 27.

[0265] In Figure 29, the reversible decoding unit 202 has an entropy decoding unit 221, which performs entropy decoding corresponding to the entropy coding performed by the entropy coding unit 181 in Figure 25.

[0266] The entropy decoding unit 221 is supplied with brightness / darkness information and encoded data from the stream expansion unit 201 (Figure 27), as well as a CLK-DET probability distribution from the encoding parameter generator 31 (Figure 10).

[0267] The entropy decoding unit 221 entropy-decodes direct entropy codes such as rANS codes and sANS codes as encoded data based on the CLK-DET probability distribution and the brightness / darkness information. In other words, the entropy decoding unit 221 selects a CLK-DET probability distribution based on the brightness / darkness information and entropy-decodes direct entropy codes such as rANS codes and sANS codes as encoded data based on the selected CLK-DET probability distribution.

[0268] For example, if the brightness information indicates that the area to be processed is a bright area, the entropy decoding unit 221 selects the CLK-DET probability distribution for bright areas from the CLK-DET probability distributions from the coding parameter generator 31, and entropy decodes the direct entropy code as encoded data using a calculation formula that utilizes that CLK-DET probability distribution for bright areas. Also, if the brightness information indicates that the area to be processed is a dark area, the entropy decoding unit 221 selects the CLK-DET probability distribution for dark areas from the CLK-DET probability distributions from the coding parameter generator 31, and entropy decodes the direct entropy code as encoded data using a calculation formula that utilizes that CLK-DET probability distribution for dark areas.

[0269] <Second configuration example of encoder 51>

[0270] Figure 30 is a block diagram showing a second configuration example of the encoder 51 in Figure 8.

[0271] In Figure 30, the encoder 51 includes a light / dark detection unit 231, a light area encoder core 232, a dark area encoder core 233, and a stream coupling unit 234.

[0272] The brightness determination unit 231 is supplied with the portion of the captured image to be processed, as well as a brightness threshold, which is one of the encoding parameters, from the encoding parameter generator 31 (Figure 10). Based on the portion to be processed and the brightness threshold, the brightness determination unit 231 determines the brightness of the portion to be processed for each pixel in the portion to be processed. For example, the brightness determination unit 231 compares the value of each pixel in the portion to be processed with the brightness threshold and determines that pixels whose pixel value is equal to or greater than the brightness threshold are bright areas. On the other hand, the brightness determination unit 231 determines that pixels whose pixel value is less than the brightness threshold are dark areas.

[0273] The brightness determination unit 231 generates brightness information, which is the result of determining the brightness of each pixel in the processing area, and supplies it to the brightness encoder core 232, the darkness encoder core 233, and the stream coupling unit 234. The brightness information generated by the brightness determination unit 231 represents the brightness of each pixel in the processing area. This brightness information is also called pixel brightness information. For example, if a bright pixel is represented by 1 and a dark pixel is represented by 0, the pixel brightness information will be a bit sequence equal to the number of pixels in the processing area.

[0274] Similar to the brightness determination unit 122 in Figure 18, the brightness determination unit 231 is supplied with camera parameters and QBI parameters, and the brightness determination unit 231 can determine the brightness of the processing target area using the camera parameters and QBI parameters as well.

[0275] The bright area encoder core 232 and the dark area encoder core 233 are supplied with brightness / darkness information from the brightness / darkness determination unit 231, as well as the processing area whose brightness / darkness is represented by that brightness / darkness information. Furthermore, the bright area encoder core 232 is supplied with the encoding parameter, which is the CLK-DET probability distribution and / or coding information, from the encoding parameter generator 31 (Figure 10) as needed.

[0276] The bright area encoder core 232 encodes the bright pixels of the processing target area using a bright area encoding scheme, based on a brightness threshold, that is, based on the pixel brightness information generated by the brightness determination unit 231 using the brightness threshold. The bright area encoder core 232 generates a bright area stream, which is a stream containing encoded data obtained by encoding the bright pixels of the processing target area, and supplies it to the stream coupling unit 234. In the bright area encoder core 232, the encoding of the bright pixels of the processing target area can also be performed based on the CLK-DET probability distribution and / or coding information, if necessary.

[0277] The dark area encoder core 233 encodes the pixels in the dark areas of the processing target using a dark area encoding scheme, based on a brightness threshold, that is, based on the pixel brightness information generated by the brightness determination unit 231 using the brightness threshold. The dark area encoder core 233 generates a dark area stream, which is a stream containing encoded data obtained by encoding the pixels in the dark areas of the processing target, and supplies it to the stream coupling unit 234. In the dark area encoder core 233, the encoding of the pixels in the dark areas of the processing target can also be performed based on the CLK-DET probability distribution and / or coding information, if necessary.

[0278] Therefore, for the area to be processed, an encoding method for bright areas or an encoding method for dark areas is selected based on the brightness of each pixel in the area to be processed, and encoding is performed using the selected encoding method.

[0279] The stream merging unit 234 appropriately merges the bright area stream from the bright area encoder core 232 and the dark area stream from the dark area encoder core 233 to generate and output an encoded stream (merged stream) in a predetermined format that incorporates the bright / dark information from the bright / dark determination unit 231.

[0280] Figure 31 is a diagram illustrating an example of pixel brightness information generated by the brightness determination unit 231 in Figure 30.

[0281] As pixel brightness information, for example, as explained in Figure 30, a bit sequence equal to the number of pixels in the area to be processed can be used, where, for example, bright pixels are represented by 1 and dark pixels are represented by 0. In this case, the number of bits for pixel brightness information will be equal to the number of pixels in the area to be processed. Therefore, for example, if the area to be processed consists of 8 pixels (8 pixels x 1 pixel), the pixel brightness information will be 8 bits.

[0282] Pixel brightness information is included in the encoded stream in the stream merging unit 234 (Figure 30), resulting in overhead. It is sometimes required to minimize this overhead.

[0283] Therefore, from all possible brightness / dark patterns of the pixels in the area to be processed, several brightness / dark patterns that are as different as possible from each other can be selected as representative patterns, and information representing the representative pattern that is most similar to the brightness / dark pattern of the pixels in the area to be processed can be adopted as pixel brightness / dark information.

[0284] Figure 31 shows an example of a typical pattern for the processing target area, which consists of 8 pixels × 1 pixel (horizontal × vertical).

[0285] In Figure 31, the 0s and 1s of each bit in the 8-bit bit sequence represent the pixels in the processing area corresponding to that bit as dark and bright pixels, respectively. In Figure 31, eight light and dark patterns are selected as representative patterns from the 2^8 possible light and dark patterns of the eight pixels in the processing area. In this case, only three bits are needed for the pixel light and dark information representing the representative patterns, which reduces the overhead of the encoded stream.

[0286] <Example configuration of the bright area encoder core 232 and the dark area encoder core 233>

[0287] Figure 32 is a block diagram showing an example configuration of the bright area encoder core 232 and dark area encoder core 233 shown in Figure 30.

[0288] In Figure 32, the bright area encoder core 232 includes a bright area acquisition unit 240, a prediction method determination unit 241, a prediction unit 242, a calculation unit 243, a quantization unit 244, an inverse quantization unit 245, a calculation unit 246, a reference buffer 247, a reversible coding unit 248, and a stream generation unit 249.

[0289] The bright area acquisition unit 240 is supplied with the processing target area, as well as pixel brightness information from the brightness / darkness determination unit 231 (Figure 30). Based on the pixel brightness / darkness information, the bright area acquisition unit 240 extracts bright pixels (pixels whose pixel brightness / darkness information represents bright areas) from the processing target area and supplies them as new processing target areas to the prediction method determination unit 241 and the calculation unit 243.

[0290] Furthermore, the bright area acquisition unit 240 can also replace the pixel values ​​of the dark area pixels in the pixel group to be processed with the pixel values ​​of the bright area pixels closest to those dark area pixels, and then supply the resulting pixel group as a new processing target to the prediction method determination unit 241 and the calculation unit 243.

[0291] The prediction method determination unit 241 to the stream generation unit 249 perform the same processing as the prediction method determination unit 151 to the stream generation unit 159 of the encoder core 123 in Figure 22. However, the reversible encoding unit 248 and the stream generation unit 249 perform some processing that differs from the reversible encoding unit 158 ​​and the stream generation unit 159 in Figure 22.

[0292] In Figure 22, the reversible coding unit 158 ​​selects an appropriate entropy code as the coding scheme for each of the processing target areas (the processing target areas determined to be bright and dark, respectively) based on the brightness information. In contrast, in Figure 32, the reversible coding unit 248, since the bright area acquisition unit 240 makes the pixels of the bright area (pixels determined to be bright) into new processing target areas, the pixels of the bright area are processed, but the pixels of the dark area (pixels determined to be dark) are not. Therefore, the reversible coding unit 248 selects an appropriate entropy code as the coding scheme for the pixels of the bright area of ​​the new processing target area.

[0293] In Figure 32, the reversible coding unit 248 is supplied with a reference image from the reference buffer 247. The reversible coding unit 248 calculates a representative value of the peripheral portion of the processing target area in the reference image from the reference buffer 247 as an illuminance value representing the brightness of the processing target area. Based on the illuminance value of the processing target area, the reversible coding unit 248 selects an appropriate entropy code as the coding method for the processing target area and performs entropy coding of the processing target area.

[0294] For example, the reversible coding unit 248 can select a code table appropriate for entropy coding of the area to be processed from multiple code tables categorized by illuminance (brightness) of the image, based on the illuminance value of the area to be processed. Multiple code tables categorized by illuminance can be supplied to the reversible coding unit 248 as code information from the coding parameter generator 31 (Figure 10), or they can be stored in the reversible coding unit 248 in advance. Alternatively, for example, the reversible coding unit 248 can select a CLK-DET probability distribution for photon count values ​​(pixel values) close to the illuminance value of the area to be processed, and then entropically code the area to be processed directly using an entropy code based on a calculation formula using the selected CLK-DET probability distribution.

[0295] The reversible coding unit 248 supplies the coded data obtained by entropy coding for the pixels of the bright areas, which are the new processing target area, to the stream generation unit 249 as bright area coded data.

[0296] In Figure 22, the stream generation unit 159 is supplied with a prediction mode, quantization width, brightness / dark information, and encoded data, and the stream generation unit 159 generates an encoded stream that includes these prediction mode, quantization width, brightness / dark information, and encoded data. In contrast, in Figure 32, the stream generation unit 249 is not supplied with brightness / dark information among the prediction mode, quantization width, brightness / dark information, and encoded data. Therefore, the stream generation unit 249 generates a stream that includes the prediction mode, quantization width, and encoded data (bright part encoded data) but does not include brightness / dark information, as a bright part stream. The stream generation unit 249 supplies the bright part stream to the stream coupling unit 234 (Figure 30).

[0297] In Figure 32, the dark area encoder core 233 includes a dark area acquisition unit 260, a reversible encoding unit 261, and a stream generation unit 262.

[0298] The dark area acquisition unit 260 is supplied with the processing target area, as well as pixel brightness information from the brightness determination unit 231 (Figure 30). Based on the pixel brightness information, the dark area acquisition unit 260 extracts dark pixels (pixels whose pixel brightness information represents a dark area) from the processing target area and supplies them to the reversible encoding unit 261 as a new processing target area.

[0299] Furthermore, the dark area acquisition unit 260 supplies the pixels (or their pixel values) of the dark areas of the processing target portion to the reference buffer 247 of the bright area encoder core 232 as a decoded dark area image.

[0300] Here, the reference buffer 247 stores as a reference image the decoded image of the original processing target area, which is reconstructed from the decoded image of the bright pixels (decoded bright area image) obtained by the arithmetic unit 246 as a new processing target area, and the pixels of the dark area (decoded dark area image) supplied from the dark area acquisition unit 260 as a new processing target area. The reconstruction of the decoded image of the original processing target area from the decoded bright area image and the decoded dark area image can be performed by referring to the pixel brightness information. The reference image stored in the reference buffer 247 is supplied to the prediction method determination unit 241, the prediction unit 242, and the reversible coding unit 248 that constitute the bright area encoder core 232, and is also supplied to the reversible coding unit 261 that constitutes the dark area encoder core 233. Therefore, it can be said that the reference buffer 247 is shared by the bright area encoder core 232 and the dark area encoder core 233.

[0301] As described above, the reversible coding unit 158 ​​in Figure 22 selects an appropriate entropy code as the coding scheme for each of the processing target areas, bright and dark, based on the brightness information. In contrast, in the reversible coding unit 261 in Figure 32, the dark area acquisition unit 260 makes the pixels in the dark area a new processing target area, so the pixels in the dark area are processed, but the pixels in the bright area are not. Therefore, the reversible coding unit 248 selects an appropriate entropy code as the coding scheme for the pixels in the dark area, which are the new processing target area.

[0302] In other words, the reversible coding unit 261, similar to the reversible coding unit 248 described above, calculates a representative value of the peripheral portion of the processing target in the reference image from the reference buffer 247 as an illuminance value representing the brightness of the processing target. Then, similar to the reversible coding unit 248, the reversible coding unit 261 selects an appropriate entropy code as the coding method for the processing target based on the illuminance value of the processing target, and performs entropy coding of the processing target.

[0303] The reversible coding unit 261 supplies the coded data obtained by entropy coding of the pixels in the dark areas, which are the new processing target areas, to the stream generation unit 262 as dark area coded data.

[0304] The stream generation unit 262 generates a stream containing dark area encoded data from the reversible encoding unit 261 as a dark area stream and supplies it to the stream coupling unit 234 (Figure 30).

[0305] In the encoder core 123 shown in Figure 22, the dark areas are quantized, resulting in lossy encoding. In contrast, the dark area encoder core 233 shown in Figure 32 does not quantize the pixels in the dark areas, resulting in lossless encoding. Therefore, in the encoder core 123 shown in Figure 22, image quality degradation in the dark areas may occur due to encoding (quantization), but in the dark area encoder core 233 shown in Figure 32, image quality degradation in the dark areas due to encoding does not occur.

[0306] Furthermore, the dark area encoder core 233 can perform lossy encoding on pixels in the dark area, similar to the encoder core 123 in Figure 22.

[0307] Figure 33 is a block diagram showing an example configuration of the reversible encoding unit 248 in Figure 32.

[0308] In Figure 33, the reversible coding unit 248 includes a table selection unit 271 and an entropy coding unit 272.

[0309] The table selection unit 271 is supplied with the peripheral portion of the processing target area of ​​the reference image from the reference buffer 247 (Figure 32), as well as coding information from the coding parameter generator 31 (Figure 10), for example, including multiple coding tables. The table selection unit 271 can be supplied with coding information including all coding tables generated by the coding parameter generator 31 (Figure 10). Furthermore, the table selection unit 271 can be supplied with coding information including coding tables from among the coding tables generated by the coding parameter generator 31 (Figure 10) that are suitable for entropy coding of bright areas, for example, coding tables generated based on the CLK-DET probability distribution for each photocount value above the brightness threshold.

[0310] The table selection unit 271 calculates a representative value of the area surrounding the area to be processed as an illuminance value representing the brightness of the area to be processed.

[0311] The table selection unit 271 selects a code table from a plurality of code tables included in the code information that is suitable for entropy coding the pixels in the bright areas of the area to be processed (more precisely, the quantized values ​​for the pixels in the bright areas) based on the illuminance value of the area to be processed. For example, it selects a code table generated based on the CLK-DET probability distribution for the photon count value closest to the illuminance value of the area to be processed, and supplies it to the entropy coding unit 272 as a selection table.

[0312] Furthermore, the table selection unit 271 can select a selection table from multiple code tables included in the code information, or it can select a selection table from any multiple code tables that have been created in advance. In other words, the table selection unit 271 can store multiple pre-created code tables and select a selection table from among those multiple code tables.

[0313] The entropy coding unit 272 is supplied with a selection table from the table selection unit 271, as well as quantization values ​​for the pixels in the bright areas of the processing target from the quantization unit 154. The entropy coding unit 272 entropy codes the quantization values ​​for the pixels in the bright areas based on the selection table, and supplies the bright area coded data obtained by this entropy coding to the stream generation unit 249 (Figure 32).

[0314] Figure 34 illustrates the generation of multiple code tables included in the code information supplied to the table selection unit 271 in Figure 33.

[0315] In the coding parameter generator 31 (Figure 10), the code designer 64, for example, designs an entropy code for each of the multiple possible values ​​of the photon count value (pixel value) based on the CLK-DET probability distribution for each of those multiple values, and generates multiple code tables in which these entropy codes are registered. As the multiple possible values ​​of the photon count value (pixel value) used to generate the code tables, for example, as in the case of Figure 24, predetermined values ​​smaller than the brightness threshold or predetermined values ​​larger than the brightness threshold can be adopted. In addition, any value such as the brightness threshold can be adopted as the multiple values. Furthermore, the multiple values ​​may be all of the possible values ​​of the photon count value (pixel value).

[0316] In Figure 34, three values, 10, 50, and 100, are adopted as multiple possible values ​​for the photon count value (pixel value). Three code tables are generated based on the CLK-DET probability distribution for each of the photon count values ​​(pixel value) = 10, 50, and 100.

[0317] If the code information includes the three code tables described above, the table selection unit 271 selects from the three code tables the code table generated from the CLK-DET probability distribution for the photon count value that is closest to the illuminance value of the processing target area as the selection table.

[0318] Note that in Figure 34, as with Figure 24, for the sake of simplicity, the photon count value (pixel value) itself is used as the random variable (horizontal axis of the CLK-DET probability distribution), and therefore the random variable of the CLK-DET probability distribution is a value of 0 or greater. When entropy coding the predicted residual (quantized value) of the processing target part, the random variable of the CLK-DET probability distribution becomes the predicted residual, and can take not only a value of 0 or greater, but also a negative value.

[0319] Figure 35 is a block diagram showing another configuration example of the reversible encoding unit 248 in Figure 32.

[0320] In Figure 35, the reversible coding unit 248 has an entropy coding unit 281.

[0321] The entropy coding unit 281 is supplied with the peripheral portion of the area to be processed from the reference image in the reference buffer 247 (Figure 32), as well as the CLK-DET probability distribution from the coding parameter generator 31 (Figure 10). Furthermore, the entropy coding unit 281 is supplied with quantization values ​​for the pixels in the bright areas of the area to be processed from the quantization unit 154.

[0322] The entropy coding unit 281 calculates a representative value of the peripheral portion of the processing target as an illuminance value representing the brightness of the processing target. Based on the CLK-DET probability distribution and the illuminance value of the processing target, the entropy coding unit 281 entropy codes the quantized values ​​of the pixels in the bright area using direct entropy codes such as rANS codes and sANS codes. That is, the entropy coding unit 281 selects a CLK-DET probability distribution based on the illuminance value of the processing target, and based on the selected CLK-DET probability distribution, entropy codes the quantized values ​​of the pixels in the bright area using direct entropy codes such as rANS codes and sANS codes.

[0323] For example, the entropy coding unit 281 selects a CLK-DET probability distribution for the photon count value that is closest to the illuminance value of the area to be processed, and then entropically codes the quantized values ​​for the pixels in the bright area directly into an entropy code using a calculation formula that utilizes the selected CLK-DET probability distribution.

[0324] The entropy coding unit 281 supplies the coded data obtained by entropy coding of the quantization values ​​for the pixels in the bright areas to the stream generation unit 249 (Figure 32) as bright area coded data.

[0325] In the dark area encoder core 233 of Figure 32, the reversible encoding unit 261 can entropy encode pixels in the dark area using a calculation formula that employs a fixed code table or a fixed probability distribution. Furthermore, the reversible encoding unit 261 can entropy encode pixels in the dark area using a code table selected based on the illuminance value of the area to be processed, or using a calculation formula that employs a CLK-DET probability distribution selected based on the illuminance value of the area to be processed, similar to the reversible encoding unit 248 of Figure 33 or Figure 35.

[0326] Figure 36 is a block diagram showing an example configuration of the reversible encoding unit 261 in Figure 32.

[0327] In Figure 36, the reversible coding unit 261 includes a table selection unit 291 and an entropy coding unit 292. Similar to the case in Figure 33, it selects a code table based on the illuminance value of the area to be processed, and then entropy codes the pixels (or their pixel values) of the dark areas from the dark area acquisition unit 250 (Figure 32) based on that code table.

[0328] The table selection unit 291 is supplied with the peripheral portion of the processing target area of ​​the reference image from the reference buffer 247 (Figure 32), as well as coding information from the coding parameter generator 31 (Figure 10), for example, including multiple coding tables. The table selection unit 291 can be supplied with coding information including all coding tables generated by the coding parameter generator 31 (Figure 10). Furthermore, the table selection unit 291 can be supplied with coding information including coding tables from among the coding tables generated by the coding parameter generator 31 (Figure 10) that are suitable for entropy coding of dark areas, for example, coding tables generated based on the CLK-DET probability distribution for each photocount value below the brightness threshold.

[0329] The table selection unit 291 calculates a representative value of the peripheral portion of the portion to be processed as an illuminance value representing the brightness of the portion to be processed. Similar to the table selection unit 271 in Figure 33, the table selection unit 291 selects a code table from among multiple code tables included in the code information, based on the illuminance value of the portion to be processed, and based on the CLK-DET probability distribution for the photon count value closest to the illuminance value of the portion to be processed, and supplies it to the entropy coding unit 292.

[0330] The entropy coding unit 292 is supplied with a selection table from the table selection unit 291, as well as pixels of the dark area to be processed from the dark area acquisition unit 250. The entropy coding unit 292 entropy codes the pixels of the dark area based on the selection table, and supplies the dark area coded data obtained by the entropy coding to the stream generation unit 262 (Figure 32).

[0331] Figure 37 is a block diagram showing another configuration example of the reversible encoding unit 261 in Figure 32.

[0332] In Figure 37, the reversible coding unit 261 has an entropy coding unit 301, and, as in the case of Figure 35, selects a CLK-DET probability distribution based on the illuminance value of the part to be processed, and entropy codes the dark area pixels from the dark area acquisition unit 250 (Figure 32) using a calculation formula that uses the CLK-DET probability distribution.

[0333] The entropy coding unit 301 is supplied with the peripheral portion of the reference image to be processed from the reference buffer 247 (Figure 32), as well as the CLK-DET probability distribution from the coding parameter generator 31 (Figure 10). Furthermore, the entropy coding unit 301 is supplied with pixels of the dark area from the dark area acquisition unit 250 (Figure 32).

[0334] The entropy coding unit 301 calculates a representative value of the peripheral portion of the processing target as an illuminance value representing the brightness of the processing target. Similar to the entropy coding unit 281 in Figure 35, the entropy coding unit 301 selects a CLK-DET probability distribution for the photon count value closest to the illuminance value of the processing target, and entropically codes the pixels in the dark areas with a direct entropy code such as a rANS code or a sANS code using a calculation formula based on the selected CLK-DET probability distribution.

[0335] The entropy coding unit 301 supplies the coded data obtained by entropy coding of the pixels in the dark areas to the stream generation unit 262 (Figure 32) as dark area coded data.

[0336] Figure 38 illustrates an example of an encoded stream generated by the stream merging unit 234 in Figure 30.

[0337] Figure 38 shows an example of the format of the unit corresponding to the portion to be processed in the encoded stream generated by the stream merging unit 234. The format in Figure 38 is also called the second format.

[0338] The unit of the second format in Figure 38 consists of a header and a payload that follows the header, similar to the first format in Figure 26.

[0339] The header includes the prediction mode and quantization width for the bright pixels of the area to be processed. Furthermore, the header includes pixel brightness information for the area to be processed.

[0340] The payload includes dark area encoded data and bright area encoded data for the portion to be processed.

[0341] Figure 39 illustrates another example of an encoded stream generated by the stream merging unit 234 in Figure 30.

[0342] Figure 39 shows another example of the format of the unit corresponding to the portion of the encoded stream generated by the stream merging unit 234 that is to be processed. The format in Figure 39 is also called the third format.

[0343] The unit of the third format in Figure 39 consists of a header and a payload that follows the header, similar to the first format in Figure 26 and the second format in Figure 38.

[0344] The header includes the prediction mode and quantization width for the pixels in the bright areas of the processing target.

[0345] The payload includes dark area encoded data and bright area encoded data containing brightness / darkness information for the portion to be processed.

[0346] The dark area encoded data containing brightness / darkness information is data that includes the brightness / darkness information of the pixels in the area to be processed and the dark area encoded data, and includes information indicating whether each pixel constituting the area to be processed is a dark area or a bright area, and, if it is a dark area pixel, the dark area encoded data of that dark area pixel.

[0347] For example, in dark area encoded data containing brightness / contrast information, the unit data consists of a sequence of bits corresponding to one pixel, with the number of units corresponding to the number of pixels in the area being processed being the same. The unit data corresponding to a dark area pixel contains the dark area encoded data of that dark area pixel, while the unit data corresponding to a bright area pixel contains information indicating that it is a bright area pixel. The information indicating that it is a bright area pixel is a value that cannot be taken from the dark area encoded data of the dark area pixel.

[0348] Figure 39 shows the dark area encoding data containing brightness / contrast information for the processing target area, which consists of 8 pixels × 1 pixel. In Figure 39, the dark area encoding data for dark pixels is represented by integers from 0 to 6, and information indicating that a pixel is a bright area is represented by 7+.

[0349] According to the dark area coding data containing brightness and darkness information described above, if the unit data is a value between 0 and 6, it is possible to identify that the pixel corresponding to that unit data is a dark area pixel. Furthermore, it is possible to identify that the value of that unit data is the dark area coding data of that dark area pixel. Furthermore, if the unit data is 7+, it is possible to identify that the pixel corresponding to that unit data is a bright area pixel. The bright area coding data of that bright area pixel is placed after the unit's dark area coding data containing brightness and darkness information.

[0350] <Second configuration example of decoder 33>

[0351] Figure 40 is a block diagram showing a second configuration example of the decoder 33 in Figure 8.

[0352] In Figure 40, the decoder 33 includes a stream splitting unit 311, a bright area decoder core 312, a dark area decoder core 313, and an image merging unit 314. The decoder 33 decodes the encoded stream obtained by encoding by the encoder 51 in Figure 30 using a decoding method corresponding to the encoding method of the encoder 51.

[0353] The stream splitting unit 311 is supplied with the encoded stream obtained by the encoding of the encoder 51 in Figure 30. The stream splitting unit 311 acquires (receives) the encoded stream and splits it to extract (split) pixel brightness information, a bright area stream, and a dark area stream from the encoded stream. The stream splitting unit 311 supplies the bright area stream to the bright area decoder core 312 and the dark area stream to the dark area decoder core 313. Furthermore, the stream splitting unit 311 supplies the pixel brightness information to the image merging unit 314.

[0354] The bright area decoder core 312 decodes the bright area stream from the stream splitting unit 311 using a decoding method for bright areas corresponding to the bright area encoding method of the bright area encoder core 232 in Figure 30. The bright area decoder core 312 supplies the decoded bright area image, which is the decoding result of the pixels (or their pixel values) of the bright area of ​​the processing target obtained by decoding, to the image merging unit 314.

[0355] The dark area decoder core 313 decodes the dark area stream from the stream splitting unit 311 using a decoding method for the dark area corresponding to the encoding method for the dark area of ​​the dark area encoder core 233 shown in Figure 30. The dark area decoder core 313 supplies the decoded dark area image, which is the decoding result of the pixels in the dark area of ​​the processing target obtained by decoding, to the image merging unit 314.

[0356] The bright area decoder core 312 and the dark area decoder core 313 are supplied with the encoding parameters, namely the CLK-DET probability distribution and / or encoding information, from the encoding parameter generator 31 (Figure 10) as needed. In the bright area decoder core 312 and the dark area decoder core 313, decoding is performed based on the CLK-DET probability distribution and / or encoding information as needed.

[0357] The image merging unit 314 merges the bright area decoded image from the bright area decoder core 312 and the dark area decoded image from the dark area decoder core 313, and outputs the decoded image of the processing target portion obtained by the merging.

[0358] <Example configuration of the bright area decoder core 312 and the dark area decoder core 313>

[0359] Figure 41 is a block diagram showing an example configuration of the bright area decoder core 312 and the dark area decoder core 313 of Figure 40.

[0360] In Figure 41, the bright area decoder core 312 includes a stream expansion unit 321, a reversible decoding unit 322, an inverse quantization unit 323, a prediction unit 324, an arithmetic unit 325, and a reference buffer 326. The bright area decoder core 312 decodes the bright area stream obtained by the encoding of the bright area encoder core 232 in Figure 30 using a bright area decoding method corresponding to the bright area encoding method of the bright area encoder core 232.

[0361] The stream expansion unit 321 is supplied with the bright area stream obtained by encoding by the bright area encoder core 232 shown in Figure 30, from the stream splitting unit 311 (Figure 40).

[0362] The stream expansion unit 321 to the reference buffer 326 perform the same processing as the stream expansion unit 201 to the reference buffer 206 of the decoder 33 in Figure 27. However, the stream expansion unit 321 and the reversible decoding unit 322 perform some processing that differs from the stream expansion unit 201 and the reversible decoding unit 202 in Figure 27.

[0363] The stream expansion unit 321 acquires and expands the bright area stream, thereby extracting the prediction mode, quantization width, and bright area encoded data of the pixels in the bright area of ​​the processing target portion from the bright area stream. The stream expansion unit 321 supplies the bright area encoded data to the reversible decoding unit 322 and the quantization width to the inverse quantization unit 323. Furthermore, the stream expansion unit 321 supplies the prediction mode to the prediction unit 324. In Figure 27, the stream expansion unit 201 handles brightness and darkness information, but the stream expansion unit 321 does not handle brightness and darkness information.

[0364] The reversible decoding unit 322 performs the same processing (reversal processing) as the reversible encoding unit 248 of the bright part encoder core 232 (Figure 32).

[0365] The reversible decoding unit 322 is supplied with bright area encoded data from the stream expansion unit 321, as well as a reference image from the reference buffer 326. Furthermore, the reversible decoding unit 322 is supplied with the CLK-DET probability distribution and / or coded information from the coding parameter generator 31 (Figure 10) as needed.

[0366] The reversible decoding unit 322 performs reversible decoding of the bright area encoded data from the stream expansion unit 321, for example, entropy decoding, and supplies the quantization values ​​for the pixels of the bright area of ​​the processing target portion, which are the result of the entropy decoding, to the inverse quantization unit 323.

[0367] The reversible decoding unit 322 performs entropy decoding corresponding to the entropy coding of the reversible coding unit 248 (Figure 32), and restores the bright area coding data, which is an entropy code obtained by the entropy coding of the reversible coding unit 248, to its original form. That is, the reversible decoding unit 322 calculates a representative value of the peripheral portion of the processing target portion in the reference image from the reference buffer 326 as an illuminance value representing the brightness of the processing target portion. Based on the illuminance value of the processing target portion, the reversible decoding unit 322 selects an appropriate entropy code as the coding method for the processing target portion and performs entropy decoding of the bright area coding data.

[0368] For example, if the reversible coding unit 248 selects a suitable code table for entropy coding of the area to be processed from multiple code tables categorized by illuminance (brightness) of the image, based on the illuminance value of the area to be processed, and performs entropy coding based on that selection table, then the reversible decoding unit 322 also selects a selection table from multiple code tables categorized by illuminance of the image, based on the illuminance value of the area to be processed, and performs entropy decoding based on that selection table, similar to the reversible coding unit 248.

[0369] Furthermore, for example, if the reversible coding unit 248 selects a CLK-DET probability distribution from the CLK-DET probability distribution supplied from the coding parameter generator 31, based on the illuminance value of the area to be processed, for the photon count value (pixel value) closest to that illuminance value, and then entropically codes the area to be processed directly into an entropy code using a calculation formula based on the selected CLK-DET probability distribution, the reversible decoding unit 322 also, similar to the reversible coding unit 248, selects a CLK-DET probability distribution from the CLK-DET probability distribution supplied from the coding parameter generator 31, based on the illuminance value of the area to be processed, for the photon count value closest to that illuminance value, and then performs entropy decoding using a calculation formula based on the selected CLK-DET probability distribution.

[0370] The inverse quantization unit 323 to the reference buffer 326 perform the same processing as the inverse quantization unit 203 to the reference buffer 206 of the decoder 33 in Figure 27 (the inverse quantization unit 245, prediction unit 242, calculation unit 246, and reference buffer 247 of the bright area encoder core 232 (Figure 32)). As a result, the calculation unit 325 obtains a bright area decoded image for the bright area pixels of the processing target portion, which is supplied to the reference buffer 326 and the image merging unit 314 (Figure 40).

[0371] In Figure 41, the dark area decoder core 313 has a stream expansion unit 331 and a reversible decoding unit 332, and decodes the dark area stream obtained by encoding by the dark area encoder core 233 in Figure 30 using a dark area decoding method corresponding to the dark area encoding method of the dark area encoder core 233.

[0372] The stream expansion unit 331 is supplied with the dark area stream obtained by encoding by the dark area encoder core 233 shown in Figure 30 from the stream splitting unit 311 (Figure 40).

[0373] The stream expansion unit 331 acquires and expands the dark area stream, thereby extracting dark area encoded data from the dark area stream. The stream expansion unit 321 supplies the dark area encoded data to the reversible decoding unit 332.

[0374] The reversible decoding unit 332 performs the same processing (reversal processing) as the reversible encoding unit 261 of the dark area encoder core 233 (Figure 32).

[0375] The reversible decoding unit 332 is supplied with dark area encoded data from the stream expansion unit 331, as well as a reference image from the reference buffer 326. Furthermore, the reversible decoding unit 332 is supplied with the CLK-DET probability distribution and / or coded information from the coding parameter generator 31 (Figure 10) as needed.

[0376] The reversible decoding unit 332 performs reversible decoding of the dark area encoded data from the stream unpacking unit 331, for example, entropy decoding, and supplies the decoded dark area image for the pixels in the dark area of ​​the processing target portion to the reference buffer 326 and the image merging unit 314 (Figure 40).

[0377] The reversible decoding unit 332 performs entropy decoding corresponding to the entropy coding of the reversible coding unit 261 (Figure 32), and restores the dark area coding data, which is an entropy code obtained by the entropy coding of the reversible coding unit 261, to its original form. That is, the reversible decoding unit 332 calculates a representative value of the peripheral portion of the processing target portion in the reference image from the reference buffer 326 as an illuminance value representing the brightness of the processing target portion. Based on the illuminance value of the processing target portion, the reversible decoding unit 322 selects an appropriate entropy code as the coding method for the processing target portion and performs entropy decoding of the dark area coding data.

[0378] Here, the reference buffer 326 stores the decoded image of the processing target area, which is reconstructed from the bright area decoded image from the calculation unit 325 and the dark area decoded image from the reversible decoding unit 332, as a reference image. Reconstruction of the decoded image of the processing target area from the bright area decoded image and the dark area decoded image can be performed by referring to the pixel brightness information. The reference image stored in the reference buffer 326 is supplied to the reversible decoding unit 322 and the prediction unit 324 that constitute the bright area decoder core 312, and is also supplied to the reversible decoding unit 332 that constitutes the dark area decoder core 313. Therefore, it can be said that the reference buffer 326 is shared by the bright area decoder core 312 and the dark area decoder core 313.

[0379] Figure 42 is a block diagram showing an example configuration of the reversible decoding unit 322 in Figure 41.

[0380] In Figure 42, the reversible decoding unit 322 includes a table selection unit 341 and an entropy decoding unit 342, and performs entropy decoding corresponding to the entropy coding performed by the reversible coding unit 248 in Figure 33.

[0381] The table selection unit 341 is supplied with the peripheral portion of the reference image to be processed from the reference buffer 326 (Figure 41), and, similar to the case of the table selection unit 271 in Figure 33, it is also supplied with coding information including multiple coding tables from the coding parameter generator 31 (Figure 10).

[0382] The table selection unit 341 calculates a representative value of the area surrounding the area to be processed as an illuminance value representing the brightness of the area to be processed.

[0383] The table selection unit 341, similar to the table selection unit 271 in Figure 33, selects a code table to be used for entropy decoding from a plurality of code tables included in the code information based on the illuminance value of the part to be processed, and supplies it to the entropy decoding unit 342 as a selected table.

[0384] In the reversible coding unit 248 shown in Figure 33, if the table selection unit 271 is to store a plurality of pre-created code tables and a selection table is to be selected from these plurality of code tables, the table selection unit 341 will also store the same plurality of code tables as those stored in the table selection unit 271, and a selection table will be selected from these plurality of code tables.

[0385] The entropy decoding unit 342 is supplied with a selection table from the table selection unit 341, as well as bright area encoded data from the stream expansion unit 321. The entropy decoding unit 342 entropy decodes the bright area encoded data based on the selection table, and supplies the quantization values ​​for the pixels in the bright area of ​​the processing target obtained by the entropy decoding to the inverse quantization unit 323 (Figure 41).

[0386] Figure 43 is a block diagram showing another example of the configuration of the reversible decoding unit 322 shown in Figure 41.

[0387] In Figure 43, the reversible decoding unit 322 has an entropy decoding unit 351, which performs entropy decoding corresponding to the entropy coding performed by the entropy coding unit 281 in Figure 35.

[0388] The entropy decoding unit 351 is supplied with bright area encoded data from the stream expansion unit 321 (Figure 41), as well as the CLK-DET probability distribution from the encoding parameter generator 31 (Figure 10). Furthermore, the entropy decoding unit 351 is supplied with the peripheral portion of the area to be processed in the reference image from the reference buffer 326 (Figure 41).

[0389] The entropy decoding unit 351 calculates a representative value of the peripheral portion of the processing target as an illuminance value representing the brightness of the processing target. Based on the CLK-DET probability distribution and the illuminance value of the processing target, the entropy decoding unit 351 entropy decodes direct entropy codes such as rANS codes and sANS codes as bright area coding data.

[0390] For example, the entropy decoding unit 351 selects a CLK-DET probability distribution for the photon count value that is closest to the illuminance value of the area to be processed, and then entropically decodes the direct entropy code as bright area encoded data using a calculation formula that utilizes the selected CLK-DET probability distribution.

[0391] The entropy decoding unit 351 supplies the quantization values ​​for the pixels in the bright areas of the processing target portion, obtained by entropy decoding of the bright area encoded data, to the inverse quantization unit 323 (Figure 41).

[0392] Figure 44 is a block diagram showing an example configuration of the reversible decoding unit 332 in Figure 41.

[0393] In Figure 44, the reversible decoding unit 332 includes a table selection unit 361 and an entropy decoding unit 362, and performs entropy decoding corresponding to the entropy coding performed by the reversible coding unit 261 in Figure 36.

[0394] The table selection unit 361 is supplied with the peripheral portion of the reference image to be processed from the reference buffer 326 (Figure 41), and, similar to the table selection unit 291 in Figure 36, is also supplied with coding information including multiple coding tables from the coding parameter generator 31 (Figure 10).

[0395] The table selection unit 361 calculates a representative value of the area surrounding the area to be processed as an illuminance value representing the brightness of the area to be processed.

[0396] The table selection unit 361, similar to the table selection unit 291 in Figure 36, selects a code table to be used for entropy decoding from a plurality of code tables included in the code information based on the illuminance value of the part to be processed, and supplies it to the entropy decoding unit 362 as a selected table.

[0397] The entropy decoding unit 362 is supplied with a selection table from the table selection unit 361, as well as dark area encoded data from the stream expansion unit 331. The entropy decoding unit 362 entropically decodes the dark area encoded data based on the selection table, and supplies the decoded dark area image for the pixels in the dark area of ​​the processing target obtained by the entropy decoding to the reference buffer 326 (Figure 41) and the image merging unit 314 (Figure 40).

[0398] Figure 45 is a block diagram showing another example of the configuration of the reversible decoding unit 332 in Figure 41.

[0399] In Figure 45, the reversible decoding unit 332 has an entropy decoding unit 371, which performs entropy decoding corresponding to the entropy coding performed by the entropy coding unit 301 in Figure 37.

[0400] The entropy decoding unit 371 is supplied with dark area encoded data from the stream expansion unit 331 (Figure 41), and, as in the case of the entropy encoding unit 301 in Figure 37, it is also supplied with a CLK-DET probability distribution from the encoding parameter generator 31 (Figure 10). Furthermore, the entropy decoding unit 371 is supplied with the peripheral portion of the processing target area of ​​the reference image from the reference buffer 326 (Figure 41).

[0401] The entropy decoding unit 371 calculates a representative value of the peripheral portion of the processing target as an illuminance value representing the brightness of the processing target. Based on the CLK-DET probability distribution and the illuminance value of the processing target, the entropy decoding unit 371 entropy decodes direct entropy codes such as rANS codes and sANS codes as dark area coding data.

[0402] For example, the entropy decoding unit 371 selects a CLK-DET probability distribution for the photon count value that is closest to the illuminance value of the area to be processed, and uses a calculation formula based on the selected CLK-DET probability distribution to entropy-decode the direct entropy code as dark area encoded data.

[0403] The entropy decoding unit 371 supplies the decoded dark area image for the pixels in the dark area of ​​the processing target portion, obtained by entropy decoding of the dark area encoded data, to the reference buffer 326 (Figure 41) and the image merging unit 314 (Figure 40).

[0404] <Another embodiment of a light sensor system applying this technology>

[0405] Figure 46 shows an example configuration of another embodiment of an optical sensor system to which this technology is applied.

[0406] In the figures, parts corresponding to those in Figure 8 are denoted by the same reference numerals, and their explanations will be omitted as appropriate below.

[0407] In Figure 46, the optical sensor system 30 includes an optical sensor 32 and a decoder 33. An encoding parameter generator 31 is configured on the logic board 42 that constitutes the optical sensor 32.

[0408] Therefore, the optical sensor system 30 in Figure 46 differs from the case in Figure 8 in that the coding parameter generator 31, which is located outside the optical sensor 32 in Figure 8, is located inside the optical sensor 32.

[0409] In Figure 46, inside the optical sensor 32, the coding parameter generator 31 generates coding parameters based on CLK-DET parameters input from an external source.

[0410] <Other examples of methods for determining the brightness threshold using the brightness criterion determination device 63>

[0411] Figure 47 illustrates another example of the method for determining the brightness threshold using the brightness criterion determination device 63 shown in Figure 10.

[0412] Figure 47, similar to Figure 13, shows the log2 graph of the standard deviation of the CLK-DET probability distribution graph in Figure 12, where the log2 of the standard deviation on the vertical axis is taken.

[0413] In the above case, we determined a single value as the brightness threshold to divide the image into either dark or bright areas. However, it is possible to determine multiple values ​​as the brightness threshold. For example, two values ​​can be determined as the brightness threshold to divide the image into three parts: dark areas, bright areas that are brighter than the dark areas, and saturated areas that are brighter than the bright areas.

[0414] For example, as explained in Figure 13, the vertical axis of the log2 graph represents the minimum number of bits required to represent a photon count value (pixel value), which is a random variable with the same units as the standard deviation. By viewing the log2 graph from the smallest photon count value, the first photon count value where the required number of bits becomes the predetermined reference number of bits can be determined as the first brightness threshold. Furthermore, the photon count value (pixel value) where the required number of bits no longer becomes the reference number of bits in the log2 graph can then be determined as the second brightness threshold.

[0415] Furthermore, for example, by pre-determining two different reference bit counts, the first brightness threshold can be determined by viewing the log2 graph from the smallest photon count value, and finding the photon count value at which the required number of bits becomes the smallest reference bit count. Subsequently, the second brightness threshold can be determined by finding the second photon count value at which the required number of bits becomes the larger reference bit count in the log2 graph.

[0416] As described above, when two values ​​are used to determine the brightness threshold, the brightness of the area to be processed is determined to be one of three categories: the darkest area, the brighter area than the dark area, or the saturated area, which is brighter than the bright area. The encoding method for the area to be processed is then selected based on the brightness of the area to be processed. In this case, it may be possible to improve the image quality of the decoded image compared to when the brightness of the area to be processed is determined to be either a dark area or a bright area.

[0417] <First modified configuration example of the encoder core 123 and decoder 33>

[0418] Figure 48 is a block diagram showing a first modified configuration example of the encoder core 123 in Figure 18.

[0419] In the figures, parts corresponding to those in Figure 22 are denoted by the same reference numerals, and their explanations will be omitted as appropriate below.

[0420] In Figure 48, the encoder core 123 includes a prediction method determination unit 151, a prediction unit 152, a calculation unit 153, a calculation unit 156, a reference buffer 157, a reversible encoding unit 158, and a stream generation unit 159. Furthermore, the encoder core 123 includes quantization units 401 and 402, and an inverse quantization unit 403.

[0421] Therefore, the encoder core 123 in Figure 48 is similar to that in Figure 22 in that it has prediction method determination units 151 to calculation units 153 and calculation units 156 to stream generation units 159. However, the encoder core 123 in Figure 48 differs from that in Figure 22 in that it does not have the quantization unit 154 and the inverse quantization unit 155 of Figure 22, and instead has newly provided quantization units 401 and 402 and an inverse quantization unit 403.

[0422] The quantization unit 401 is supplied with the portion to be processed. Similar to the quantization unit 154 in Figure 22, the quantization unit 401 quantizes each pixel (or its pixel value) of the portion to be processed and supplies the resulting quantized values ​​to the prediction method determination unit 151 and the calculation unit 153. Therefore, in Figure 22, the prediction residual is quantized, but in Figure 48, each pixel of the portion to be processed is quantized first, and the prediction residual is generated using the pixel values ​​that have become quantized.

[0423] Therefore, the quantization unit 402 is supplied with a reference image from the reference buffer 157, and the quantization unit 402 quantizes the reference image and supplies it to the prediction method determination unit 151 and the prediction unit 152. As a result, the prediction method determination unit 151 determines the prediction mode and the quantization width to be used when performing quantization, based on the portion to be processed after quantization and the reference image after quantization. The prediction unit 152 then uses the reference image after quantization to predict the predicted image of the portion to be processed after quantization.

[0424] Furthermore, in Figure 48, the output of the arithmetic unit 156 is the decoded image of the processed portion after quantization, which is obtained by adding the predicted residual for the processed portion after quantization and the predicted image of the processed portion after quantization.

[0425] Therefore, the decoded image of the processing target portion after quantization, which is the output of the arithmetic unit 156, is supplied to the inverse quantization unit 403. The inverse quantization unit 403 generates a decoded image of the processing target portion by inverse quantization of the decoded image of the processing target portion after quantization, and stores it in the reference buffer 157 as a reference image.

[0426] The bright area encoder core 232 in Figure 32 can be configured in the same way as the encoder core 123 in Figure 48.

[0427] Figure 49 is a block diagram showing a first modified configuration example of the decoder 33 in Figure 8.

[0428] In other words, Figure 49 shows an example of the configuration of the decoder 33 when the encoder core 123 is configured as shown in Figure 48.

[0429] In the figures, parts corresponding to those in Figure 27 are denoted by the same reference numerals, and their explanations will be omitted as appropriate below.

[0430] In Figure 49, the decoder 33 includes a stream expansion unit 201, a reversible decoding unit 202, a prediction unit 204, an arithmetic unit 205, and a reference buffer 206. Furthermore, the decoder 33 includes an inverse quantization unit 411 and a quantization unit 412.

[0431] Therefore, the decoder 33 in Figure 49 is similar to that in Figure 27 in that it has a stream expansion unit 201, a reversible decoding unit 202, and prediction units 204 to a reference buffer 206. However, the decoder 33 in Figure 49 differs from that in Figure 27 in that it does not have an inverse quantization unit 203, and an inverse quantization unit 411 and a quantization unit 412 are newly provided.

[0432] In Figure 49, the reversible decoding unit 202 supplies the calculation unit 205 with the predicted residual for the portion to be processed after quantization. The calculation unit 205 adds the predicted image of the portion to be processed after quantization, supplied by the prediction unit 204, to the predicted residual for the portion to be processed after quantization, and generates a decoded image of the portion to be processed after quantization.

[0433] In the arithmetic unit 205, a decoded image of the processing target portion after quantization is generated, and therefore, the decoded image needs to be dequantized. For this reason, the decoded image of the processing target portion after quantization is supplied from the arithmetic unit 205 to the dequantization unit 411. The dequantization unit 411 generates a decoded image of the processing target portion by dequantizing the decoded image of the processing target portion after quantization, and stores it in the reference buffer 206 as a reference image.

[0434] The quantization unit 412 is supplied with a reference image from the reference buffer 157. The quantization unit 412 quantizes the reference image and supplies it to the prediction unit 204. As a result, the prediction unit 204 uses the quantized reference image to predict the predicted image of the part to be processed after quantization and supplies it to the calculation unit 205.

[0435] The bright area decoder core 312 in Figure 41 can be configured in the same way as the decoder 33 in Figure 49.

[0436] <Second modified configuration example of the encoder core 123 and decoder 33>

[0437] Figure 50 is a block diagram showing a second modified configuration example of the encoder core 123 in Figure 18.

[0438] In the figures, parts corresponding to those in Figure 22 are denoted by the same reference numerals, and their explanations will be omitted as appropriate below.

[0439] In Figure 50, the encoder core 123 includes a prediction method determination unit 151, a prediction unit 152, a calculation unit 153, a calculation unit 156, a reference buffer 157, a reversible encoding unit 158, and a stream generation unit 159. Furthermore, the encoder core 123 includes a spatial transformation quantization unit 421 and an inverse quantization inverse spatial transformation unit 422.

[0440] Therefore, the encoder core 123 in FIG. 50 is common to the case of FIG. 22 in that it includes a prediction method determination unit 151 to a calculation unit 153, and a calculation unit 156 to a stream generation unit 159. However, the encoder core 123 in FIG. 50 is different from the case of FIG. 22 in that it includes a spatial transform quantization unit 421 and an inverse quantization inverse spatial transform unit 422 instead of the quantization unit 154 and the inverse quantization unit 155 in FIG. 22.

[0441] The spatial transform quantization unit 421 is supplied with a prediction residual from the calculation unit 153. The spatial transform quantization unit 421 performs an orthogonal transform of the prediction residual from the calculation unit 153 into the frequency space, for example, a DCT (discrete cosine transform). Further, the spatial transform quantization unit 421 quantizes the transform coefficients obtained by the orthogonal transform of the prediction residual, and supplies the resulting quantization values to the reversible encoding unit 158 and the inverse quantization inverse spatial transform unit 422.

[0442] The inverse quantization inverse spatial transform unit 422 inverse quantizes the quantization values from the spatial transform quantization unit 421. Further, the inverse quantization inverse spatial transform unit 422 performs an inverse (orthogonal) transform of the transform coefficients obtained by the inverse quantization from the frequency space, for example, an inverse DCT, and supplies the resulting prediction residual to the calculation unit 156.

[0443] The bright part encoder core 232 in FIG. 32 can also be configured in the same manner as the encoder core 123 in FIG. 50.

[0444] FIG. 51 is a block diagram showing a second modified configuration example of the decoder 33 in FIG. 8.

[0445] That is, FIG. 51 shows a configuration example of the decoder 33 when the encoder core 123 is configured as shown in FIG. 50.

[0446] In the figure, parts corresponding to the case of FIG. 27 are denoted by the same reference numerals, and the description thereof will be omitted as appropriate below.

[0447] In FIG. 51, the decoder 33 includes a stream expansion unit 201, a reversible decoding unit 202, a prediction unit 204, a calculation unit 205, and a reference buffer 206. Further, the decoder 33 includes an inverse quantization inverse spatial transform unit 431.

[0448] Therefore, the decoder 33 in FIG. 51 is common to the case of FIG. 27 in that it includes a stream expansion unit 201, a reversible decoder unit 202, a prediction unit 204 to a reference buffer 206. However, the decoder 33 in FIG. 51 is different from the case of FIG. 27 in that an inverse quantization inverse spatial transformation unit 431 is provided instead of the inverse quantization unit 203.

[0449] In FIG. 51, a quantized value obtained by quantizing the transform coefficient of the prediction residual is supplied from the reversible decoder unit 202 to the inverse quantization inverse spatial transformation unit 431. The inverse quantization inverse spatial transformation unit 431 inverse-quantizes the quantized value from the reversible decoder unit 202. Further, the inverse quantization inverse spatial transformation unit 422 performs, for example, an inverse DCT as an inverse transformation from the frequency space of the transform coefficient obtained by the inverse quantization, and supplies the resulting prediction residual to the arithmetic unit 205.

[0450] The bright part decoder core 312 in FIG. 41 can also be configured in the same manner as the decoder 33 in FIG. 51.

[0451] <Other Configuration Example of Bright Part Encoder Core 232>

[0452] FIG. 52 is a block diagram showing another configuration example of the bright part encoder core 232 in FIG. 30.

[0453] In the figure, parts corresponding to the case of FIG. 32 are denoted by the same reference numerals, and the description thereof will be omitted as appropriate below.

[0454] In FIG. 52, the bright part encoder core 232 includes a bright part acquisition unit 240 to an arithmetic unit 243, and an inverse quantization unit 245 to a stream generation unit 249. Further, the bright part encoder core 232 includes a quantization unit 441.

[0455] Therefore, the bright part encoder core 232 in FIG. 52 is common to the case of FIG. 32 in that it includes a bright part acquisition unit 240 to an arithmetic unit 243, and an inverse quantization unit 245 to a stream generation unit 249. However, the bright part encoder core 232 in FIG. 52 is different from the case of FIG. 32 in that a quantization unit 441 is provided instead of the quantization unit 244.

[0456] The quantization unit 441 is supplied with the predicted residuals of the pixels in the bright areas of the processing target from the calculation unit 243. Furthermore, the quantization unit 441 is supplied with a reference image from the reference buffer 247 and a CLK-DET probability distribution from the coding parameter generator 31 (Figure 10).

[0457] The quantization unit 441 sets a quantization width based on a CLK-DET probability distribution selected from the CLK-DET probability distribution from the coding parameter generator 31 (Figure 10) based on the reference image from the reference buffer 247, and quantizes the predicted residual from the calculation unit 243 with that quantization width.

[0458] Figure 53 is a block diagram showing an example configuration of the quantization unit 441 in Figure 52.

[0459] In Figure 53, the quantization unit 441 includes a representative value acquisition unit 451, a quantization width determination unit 452, and a quantization unit 453.

[0460] The representative value acquisition unit 451 is supplied with a reference image from the reference buffer 247 (Figure 52). The representative value acquisition unit 451 calculates a representative value of the peripheral portion of the processing target portion in the reference image from the reference buffer 247 as an illuminance value representing the brightness of the processing target portion, and supplies it to the quantization width determination unit 452.

[0461] The quantization width determination unit 452 is supplied with illuminance values ​​of the processing target area from the representative value acquisition unit 451, as well as a CLK-DET probability distribution from the coding parameter generator 31 (Figure 10). The quantization width determination unit 452 selects a CLK-DET probability distribution from the CLK-DET probability distribution from the coding parameter generator 31 (Figure 10) based on the illuminance values ​​of the processing target area from the representative value acquisition unit 451. For example, the quantization width determination unit 452 selects a CLK-DET probability distribution for the photon count value closest to the illuminance value of the processing target area. Based on the selected CLK-DET probability distribution, the quantization width determination unit 452 adjusts the quantization width determined by the prediction method determination unit 241, determines (sets) the adjusted quantization width as the final quantization width to be used for quantization, and supplies it to the quantization unit 453.

[0462] The quantization unit 453 is supplied with a quantization width from the quantization width determination unit 452, as well as predicted residuals of pixels in the bright areas of the processing target area from the calculation unit 243 (Figure 52). The quantization unit 453 quantizes the predicted residuals from the calculation unit 243 using the quantization width from the quantization width determination unit 452, and supplies the resulting quantized values ​​to the inverse quantization unit 245 and the reversible coding unit 248 (Figure 52).

[0463] In the inverse quantization unit 245, inverse quantization is performed using the quantization width adjusted in the quantization unit 441 (specifically, the quantization width determination unit 452).

[0464] Furthermore, when the bright area encoder core 232 is configured as shown in Figure 52, the encoded stream can include the quantization width after adjustment by the quantization unit 441, or the quantization width before adjustment (the quantization width determined by the prediction method determination unit 241 (Figure 32)).

[0465] If the encoded stream includes the adjusted quantization width from the quantization unit 441, the inverse quantization unit 323 of the bright part decoder core 312 in Figure 41 performs inverse quantization using the adjusted quantization width included in the encoded stream.

[0466] If the encoded stream includes the quantization width before adjustment by the quantization unit 441, the inverse quantization unit 323 of the bright part decoder core 312 in Figure 41 adjusts the quantization width included in the encoded stream in the same manner as the quantization unit 441. The inverse quantization unit 323 then performs inverse quantization with the adjusted quantization width.

[0467] Figure 54 illustrates an example of adjusting the quantization width using the quantization width determination unit 452 in Figure 53.

[0468] Figure 54 shows an example of a selected probability distribution (a CLK-DET probability distribution selected for a selection).

[0469] The quantization width determination unit 452 selects the CLK-DET probability distribution from the coding parameter generator 31 to the illuminance value of the processing target area from the representative value acquisition unit 451, and uses that CLK-DET probability distribution for the photon count value (pixel value) closest to that illuminance value as the selection probability distribution. The quantization width determination unit 452 adjusts the quantization width determined by the prediction method determination unit 241 based on the variance σ^2 of the selection probability distribution. For example, the quantization width determination unit 452 adjusts the quantization width determined by the prediction method determination unit 241 by multiplying the quantization width determined by the prediction method determination unit 241 by a value based on the variance σ^2 of the selection probability distribution.

[0470] As a value based on the variance σ² of the selection probability distribution, for example, a value proportional to the variance σ² of the selection probability distribution can be adopted. In this case, the smaller (larger) the variance σ² of the selection probability distribution, the smaller (larger) the quantization width is adjusted to. As a result, for example, the part of the data to be processed that is likely to have a small variance is quantized with a small quantization width, and the part of the data to be processed that is likely to have a large variance is quantized with a large quantization width.

[0471] <Examples of applications for the optical sensor system 30>

[0472] Figure 55 is a block diagram showing an example configuration of a signal processing system to which the optical sensor system 30 of Figure 8 or Figure 46 is applied.

[0473] In Figure 55, the signal processing system 500 includes an optical sensor 32, a real-time playback system 511, a DB (database) 512, an on-demand playback system 513, and a cloud AI system 514.

[0474] In the optical sensor 32, the sensor board 41 performs photoelectric conversion at the pixels, acquires pixel values ​​using the CLK-DET method, and outputs them to the logic board 42.

[0475] The logic board 42 has an encoder 51 and a signal processing unit 501 formed on it. The signal processing unit 501 performs necessary signal processing on the captured image, which is composed of pixel values ​​acquired by the CLK-DET method on the sensor board 41. The encoder 51 compresses the captured image, which is the RAW data after signal processing by the signal processing unit 501, based on encoding parameters from an encoding parameter generator 31 (Figure 10), which is not shown in Figure 55, and outputs the encoded stream obtained by this encoding to the outside of the optical sensor 32, for example, to the real-time playback system 511 and DB 512.

[0476] In the encoder 51, the captured image is divided into, for example, bright and dark areas based on encoding parameters generated based on the CLK-DET parameters. The bright areas are encoded using an encoding method for bright areas, and the dark areas are encoded using an encoding method for bright areas. Therefore, the captured image can be compressed while suppressing degradation of image quality. In particular, for captured images composed of pixel values ​​acquired by the CLK-DET method, as explained in Figure 7, if the captured image is encoded using an existing encoding method, the image quality of the dark areas will be greatly degraded due to the fact that many bits (gradations) are assigned to the dark areas. In the encoder 51, an encoding method that suppresses degradation of image quality in the dark areas (or a lossless encoding method) is adopted as the encoding method for the dark areas, so that the captured image can be compressed while suppressing degradation of image quality in the dark areas, and compression suitable for captured images composed of pixel values ​​acquired by the CLK-DET method can be performed.

[0477] As a result, while suppressing the degradation of image quality in the dark areas of the captured image, the transmission bandwidth of the transmission IF that outputs data to external devices such as the real-time playback system 511 or DB512 can be reduced. For example, the transmission bandwidth of the transmission IF can be reduced to about half or less compared to outputting the captured image as RAW data. For example, as explained in Figure 50, when performing an orthogonal transformation of the predicted residual into the frequency space and quantizing the transformation coefficients obtained by that orthogonal transformation, the transmission bandwidth of the transmission IF can be reduced to about 1 / 10.

[0478] By reducing the transmission band of the transmission IF, the power consumption can be reduced. Also, when the transmission IF is a wired IF, the number of physical wires constituting the wired IF can be reduced. Further, in the encoded stream obtained by encoding the captured image, the amount of data per frame is smaller compared to the captured image, so the transmission frame rate can be increased compared to the case of outputting the captured image as it is.

[0479] The real-time playback system 511 includes a decoder 521, a developing processing unit 522, and a display unit 523, and performs real-time playback of the captured image.

[0480] The decoder 521 is configured in the same manner as the decoder 33, decodes the encoded stream from the optical sensor 32, and supplies the decoded image of the RAW data obtained by the decoding to the developing processing unit 522. Note that necessary encoding parameters are supplied to the decoder 521 from an encoding parameter generator 31 (FIG. 10) not shown in FIG. 55. The decoder 521 decodes the encoded stream based on the necessary encoding parameters in the same manner as the decoder 33. The same applies to the decoders 531 and 541 described later.

[0481] The developing processing unit 522 develops the decoded image of the RAW data from the decoder 521, and supplies the developed decoded image to the display unit 523.

[0482] The display unit 523 displays the decoded image from the developing processing unit 522.

[0483] The DB 512 stores the encoded stream from the optical sensor 32.

[0484] The encoded stream stored in the DB 512 is transmitted to the on-demand playback system 513 as needed and / or uploaded to the cloud AI system 514 via the network.

[0485] Regarding the DB 512, the storage capacity can be reduced compared to the case of storing the captured image as it is. Also, the transmission band between the DB 512 and each of the on-demand playback system 513 and the cloud AI system 514 can be reduced.

[0486] The on-demand playback system 513 includes a decoder 531, a development processing unit 532, and a display unit 533, and performs on-demand playback of captured images.

[0487] The decoder 531 is configured similarly to the decoder 33, decodes the encoded stream from the DB 512, and supplies the decoded image of the RAW data obtained by the decoding to the development processing unit 532.

[0488] The development processing unit 532 develops the decoded image of the RAW data from the decoder 531 and supplies the developed decoded image to the display unit 533.

[0489] The display unit 533 displays the decoded image from the development processing unit 532.

[0490] The cloud AI system 514 includes a decoder 541 and an AI processing unit 542.

[0491] The decoder 541 is configured similarly to the decoder 33, decodes the encoded stream from DB 512, and supplies the decoded image of the RAW data obtained by the decoding to the AI ​​processing unit 542.

[0492] The AI ​​processing unit 542 uses the decoded image of the RAW data from the decoder 541 as training data to train the AI ​​model. Alternatively, the AI ​​processing unit 542 provides the decoded image of the RAW data from the decoder 541 to the AI ​​model as a prompt or the like to generate image or other information.

[0493] Figure 56 is a block diagram showing an example configuration of another signal processing system to which the optical sensor system 30 of Figure 8 or Figure 46 is applied.

[0494] In Figure 56, the signal processing system 600 includes an optical sensor 32, a QBI signal processing unit 611, DRAMs (dynamic random access memory) 612, 613, 614, encoders 621, 622, 623, and decoders 631, 632, 633, 634.

[0495] In the optical sensor 32, as in the case of Figure 55, an image composed of pixel values ​​acquired by the CLK-DET method is captured (acquired). However, in Figure 56, the image is captured by BI. In the optical sensor 32, as in the case of Figure 55, the signal processing unit 501 performs the necessary signal processing on the captured image. Furthermore, as in the case of Figure 55, the encoder 51 performs encoding to compress the captured image after signal processing by the signal processing unit 501, based on encoding parameters from the encoding parameter generator 31 (Figure 10), which is not shown in Figure 56. The encoded stream obtained by encoding by the encoder 51 is transmitted to the QBI signal processing unit 611, which is outside the optical sensor 32.

[0496] Since an encoded stream is transmitted from the optical sensor 32 to the QBI signal processing unit 611, the transmission bandwidth between the optical sensor 32 and the QBI signal processing unit 611 can be reduced compared to when the captured image itself is transmitted.

[0497] The QBI signal processing unit 611 includes a decoder 650, an addition processing unit 651, a motion estimation unit 652, a motion vector interpolation unit 653, a non-blurred image generation unit 654, and a 3DNR (3-dimensional noise reduction) unit 655.

[0498] The decoder 650 is supplied with an encoded stream from the optical sensor 32. The decoder 650 is configured similarly to the decoder 33, decodes the encoded stream from the optical sensor 32, and supplies the decoded image obtained from the decoding to the addition processing unit 651 and the encoder 622. The decoder 650 is supplied with the necessary encoding parameters from an encoding parameter generator 31 (Figure 10), which is not shown in Figure 56. The decoder 650 decodes the encoded stream based on the necessary encoding parameters, similar to the decoder 33. The decoders 631 to 634, which will be described later, are configured similarly.

[0499] The encoder 622 encodes the decoded image from the decoder 650 and supplies the resulting encoded stream to the DRAM 613 for storage. The encoder 622 performs encoding based on encoding parameters from the encoding parameter generator 31 (Figure 10), which is not shown in Figure 56. The other encoders 621 and 623 operate similarly.

[0500] The encoded stream stored in the DRAM 613 is supplied to the decoder 633 as needed. The decoder 633 decodes the encoded stream from the DRAM 613 and supplies the decoded image obtained from the decoding to the non-blurred image generation unit 654 of the QBI signal processing unit 611.

[0501] As described above, since encoded data is stored in the DRAM 613, the input / output transmission bandwidth and storage capacity of the DRAM 613 can be reduced compared to the case where the captured image (or its decoded image) is stored.

[0502] Furthermore, the DRAM 613 can store the encoded stream supplied from the encoder 51 to the decoder 650, rather than the encoded stream encoded by the encoder 622 from the decoder 650. In this case, the encoder 622 is not required.

[0503] Furthermore, a decoded image obtained by encoding and decoding a captured image (including an image obtained by encoding and decoding a decoded image) is identical to the captured image, provided that the degradation of image quality due to encoding and decoding is not considered. Therefore, in the following explanation, for simplicity's sake, the decoded image will also be referred to as the captured image.

[0504] The addition processing unit 651 is supplied with the captured image, which is the decoded image from the decoder 650, as well as the added image from the decoder 631. The addition processing unit 651 adds the captured image from the decoder 650 to the added image from the decoder 631, and supplies the new added image generated by this addition to the encoder 621. The added image is an image obtained by adding multiple captured images in a time series without alignment, and therefore has a certain degree of motion blur and noise.

[0505] The encoder 621 encodes the summation image from the addition processing unit 651 and supplies the resulting encoded stream to the DRAM 612 for storage. The encoded stream stored in the DRAM 612 is supplied to the decoders 631 and 632 as needed. The decoders 631 and 632 decode the encoded stream from the DRAM 612. The decoder 631 supplies the summation image obtained by decoding to the addition processing unit 651 of the QBI signal processing unit 611, and the decoder 632 supplies the summation image obtained by decoding to the motion estimation unit 652 of the QBI signal processing unit 611.

[0506] As described above, since encoded data is stored in the DRAM 612, the input / output transmission bandwidth and storage capacity of the DRAM 612 can be reduced compared to the case where added images are stored.

[0507] The motion estimation unit 652 estimates the motion of the summed image from the decoder 632 and supplies the summed image motion vector representing that motion to the motion vector interpolation unit 653 and the 3DNR unit 655.

[0508] The motion vector interpolation unit 653 generates original image motion vectors representing the motion of each of the multiple captured images that were added together in the generation of an added image representing motion, based on the added image motion vector from the motion estimation unit 652, and supplies these to the non-blurred image generation unit 654.

[0509] The non-blurred image generation unit 654 superimposes (adds) the captured images, which are decoded images from the decoder 633, while aligning them according to the motion vectors of the original images from the motion vector interpolation unit 653, which represent the motion of the captured images. By superimposing multiple captured images from the decoder 633 while aligning them, the non-blurred image generation unit 654 generates a non-blurred image with no (reduced) motion blur at high DR and supplies it to the 3DNR unit 655.

[0510] The 3DNR unit 655 is supplied with a non-blurred image from the non-blurred image generation unit 654, as well as an additive image motion vector from the motion estimation unit 652, and a denoised image from the decoder 634. The 3DNR unit 655 performs 3DNR processing on the non-blurred image from the non-blurred image generation unit 654 based on the denoised image from the decoder 634 and the additive image motion vector from the motion estimation unit 652. The 3DNR unit 655 supplies the new denoised image obtained by the 3DNR processing, which is obtained by removing (reducing) noise from the non-blurred image from the non-blurred image generation unit 654, to the encoder 623 and outputs it as the result of QBI signal processing.

[0511] The encoder 623 encodes the denoised image from the 3DNR unit 655 and supplies the encoded stream obtained by this encoding to the DRAM 614 for storage. The encoded stream stored in the DRAM 614 is supplied to the decoder 634 as needed. The decoder 634 decodes the encoded stream from the DRAM 614. The decoder 634 supplies the denoised image obtained by decoding to the 3DNR unit 655 of the QBI signal processing unit 611.

[0512] As described above, since encoded data is stored in the DRAM 614, the input / output transmission bandwidth and storage capacity of the DRAM 614 can be reduced compared to the case where the denoised image is stored.

[0513] By employing an encoder similar to the encoder 51 and a decoder similar to the decoder 33 in the IF of the memory that stores data subject to arbitrary signal processing, not limited to the QBI signal processing performed by the QBI signal processing unit 611, the transmission bandwidth of the memory's input / output and the storage capacity can be reduced.

[0514] In Figure 56, for the three DRAMs 612 to 614, fewer than that number of DRAMs, i.e., one or two DRAMs, can be used as DRAMs 612 to 614. Similarly, for the three encoders 621 to 623, fewer than that number of encoders can be used as encoders 621 to 623. Similarly, for the five decoders 650 and 631 to 634, fewer than that number of decoders can be used as decoders 650 and 631 to 634.

[0515] <Description of a computer using this technology>

[0516] The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up that software are installed on a computer. Here, a computer includes computers built into dedicated hardware, as well as general-purpose personal computers, for example, that can perform various functions by installing various programs.

[0517] Figure 57 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above using a program.

[0518] In a computer, the processing circuit 901, ROM (Read Only Memory) 902, and RAM (Random Access Memory) 903 are interconnected by a bus 904.

[0519] An input / output interface 905 is further connected to the bus 904. An input unit 906, an output unit 907, a storage unit 908, a communication unit 909, and a drive 910 are connected to the input / output interface 905.

[0520] The input unit 906 may include physical or virtual operating means that the user operates to input information, such as a keyboard, mouse, or touch panel, as well as means that the user inputs information through voice, eye gaze, etc. Furthermore, the input unit 906 may include sensors for inputting various physical quantities to the computer. For example, the input unit 906 may include sensors that acquire physical quantities such as light (including infrared light other than visible light) or sound, such as a camera or microphone. Also, for example, the input unit 906 may include sensors that acquire other physical quantities such as temperature, moisture content, acceleration, distance, etc. The output unit 907 may include means that present information to the user by stimulating the user's perception, such as a display, speaker, or haptic device. The storage unit 908 is composed of a hard disk, non-volatile or volatile memory, etc., and stores various types of information (including programs). The communication unit 909 is a network interface, etc., and performs wired or wireless communication with the outside. The drive 910 drives removable media 911 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0521] The processing circuit 901 includes a processor that executes programs such as a CPU (Central Processing Unit) and a DSP (Digital Signal Processor). The processing circuit 901 (its processor) performs the above-described series of processes by loading the program stored in the storage unit 908 into the RAM 903 via the input / output interface 905 and the bus 904 and executing it. The processing circuit 901 can output the processing results of the series of processes from the output unit 907 via the bus 904 and the input / output interface 905 as needed. The processing circuit 901 can also store the processing results in the storage unit 908 or transmit them from the communication unit 909.

[0522] The program executed by the computer (processing circuit 901) can be provided by recording it on a removable medium 911, such as a package medium. The program can also be provided via wired or wireless transmission media, such as a local area network, the internet, or digital satellite broadcasting.

[0523] In a computer, a program can be installed in the storage unit 908 via the input / output interface 905 by inserting a removable media 911 into the drive 910. Alternatively, a program can be received by the communication unit 909 from another device, such as a server, via a wired or wireless transmission medium, and installed in the storage unit 908. Furthermore, programs can be pre-installed in the ROM 902 or the storage unit 908.

[0524] The programs executed by the computer may be programs that are processed chronologically in the order described herein, or they may be programs that are processed in parallel or at necessary times, such as when a call is made.

[0525] The processes that a computer performs according to a program do not necessarily have to follow the order described in the flowchart. In other words, the processes that a computer performs according to a program include processes that are executed in parallel or individually (e.g., parallel processing and object-based processing).

[0526] The program may be processed by a single computer (processor), or it may be processed in a distributed manner by multiple computers. Furthermore, the program may be transferred to a remote computer and executed there.

[0527] When the above-described series of processes are performed by a computer executing a program, the processing circuit 901 (its processor) functions as an encoding parameter generator 31, a decoder 33, and an encoder 51 by executing the program.

[0528] In this specification, a system means one component or a collection of multiple components (devices, modules (parts), etc.). Therefore, one or more components of a computer, for example, only the processor, or a combination of the processor and memory (for example, only the processing circuit 901, or a combination of the processing circuit 901 to the bus 904, etc.), constitute a system. Regarding a collection of multiple components, it is not necessary whether all components reside in the same enclosure. Therefore, multiple devices housed in separate enclosures and connected via a network, or a single device containing multiple modules within a single enclosure, are all systems. Furthermore, for example, the entire computer, or a combination of a computer and other devices such as a server (not shown), also constitute a system.

[0529] Furthermore, the embodiments of this technology are not limited to those described above, and various modifications are possible without departing from the spirit of this technology.

[0530] For example, this technology can be configured as cloud computing, where a single function is shared and processed collaboratively by multiple devices via a network.

[0531] Furthermore, each step described in the flowchart above can be performed by a single device, or it can be divided and performed by multiple devices.

[0532] Furthermore, if a single step includes multiple processes, those processes can be executed by a single device or shared among multiple devices.

[0533] Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may also occur.

[0534] Furthermore, this technology can take the following configuration.

[0535] <1> An encoder that encodes a captured image composed of the pixel values ​​acquired by a CLK-DET (clocked detector) based on encoding parameters generated based on CLK-DET parameters of a CLK-DET (clocked detector) method, in which the maximum count value of photons is determined by counting the number of periods of the counting period in the exposure time, and the pixel values ​​are acquired based on the count value of the photons counted. <2> The encoder according to <1>, wherein the encoding parameters include a brightness threshold used to determine the brightness of the processing target portion of the captured image, and the encoding method for the processing target portion is selected based on the brightness of the processing target portion determined based on the brightness of the brightness of the processing target portion. <3> The encoder according to <2>, wherein the brightness threshold is adjusted based on camera parameters when capturing the captured image, and / or parameters for signal processing on the captured image acquired by burst imaging. <4> The encoder according to <2> or <3>, wherein the brightness of the processing target portion is determined based on the brightness threshold and representative values ​​of the pixel values ​​of the processing target portion and / or the peripheral portion of the processing target portion. <5> An encoder according to <2> or <3> that determines the brightness or darkness of each pixel in the processing target portion. <6> An encoder according to any one of <1> to <5>, wherein the encoding parameter includes code information relating to an entropy code, and based on the code information, entropy encodes the processing target portion of the captured image. <7> An encoder according to <6> that selects an entropy code to entropy encode the processing target portion based on the processing target portion and / or the surrounding portion of the processing target portion. <8> An encoder according to any one of <1> to <7>, wherein the encoding parameter includes a CLK-DET probability distribution in which photons are incident during the count period in the exposure time, and based on the CLK-DET probability distribution, encodes the processing target portion of the captured image. <9> An encoder according to <8> that sets the quantization width for encoding the processing target portion based on the CLK-DET probability distribution.<10> A decoder that decodes encoded data encoded by an encoding method which encodes a captured image composed of the pixel values ​​acquired by a CLK-DET (clocked detector) based on encoding parameters generated based on CLK-DET parameters of a CLK-DET (clocked detector) method in which the maximum count value of photons is determined by counting the number of periods of the counting period in the exposure time, and the pixel values ​​are acquired based on the count value of the photons counted. <11> The decoder according to <10>, wherein the encoding parameter includes a brightness threshold used to determine the brightness of the processing target portion of the captured image, and the decoder selects the encoding method for the processing target portion based on the brightness of the processing target portion determined based on the brightness threshold. <12> The decoder according to <11>, wherein the brightness threshold is adjusted based on camera parameters used when capturing the captured image and / or parameters for signal processing on the captured image acquired by burst imaging. <13> The decoder according to <11> or <12>, wherein the decoding method determines the brightness of the processing target portion based on the brightness threshold and representative values ​​of the pixel values ​​of the processing target portion and / or the peripheral portion of the processing target portion. <14> The decoder according to <11> or <12>, wherein the decoding method determines the brightness of each pixel of the processing target portion. <15> The decoder according to any one of <10> to <14>, wherein the decoding parameter includes code information relating to an entropy code, and the decoder entropically encodes the processing target portion of the captured image based on the code information. <16> The decoder according to <15>, wherein the decoding method selects an entropy code to entropically encode the processing target portion based on the processing target portion and / or the peripheral portion of the processing target portion.<17> A decoder according to any one of <10> to <16>, wherein the encoding parameter includes a CLK-DET probability distribution in which photons are incident during the count period in the exposure time, and the decoder encodes the portion of the captured image to be processed based on the CLK-DET probability distribution. <18> A decoder according to <17>, wherein the decoding method sets the quantization width for encoding the portion to be processed based on the CLK-DET probability distribution. <19> An optical sensor comprising a plurality of pixels that perform photoelectric conversion, and an encoder that encodes a captured image composed of the pixel values ​​obtained by the CLK-DET (clocked detector) method, based on an encoding parameter generated based on a CLK-DET parameter of the CLK-DET method, in which the maximum count value of photons is determined and the number of periods in the count period in the exposure time is determined and the pixel values ​​are obtained based on the count value of the photons. <20> The optical sensor according to <19>, wherein the CLK-DET parameters are the length of one or more of the counting periods and the number of periods for each of the counting periods, and further includes an encoding parameter generator that generates the encoding parameters based on the CLK-DET parameters.

[0536] 10 Signal processing system, 11 Optical sensor, 12 QBI signal processing unit, 13 Development processing unit, 14 Display unit, 15 Encoding unit, 16 Storage medium, 30 Optical sensor system, 31 Encoding parameter generator, 32 Optical sensor, 33 Decoder, 41 Sensor board, 42 Logic board, 51 Encoder, 61 Probability distribution calculator, 62 Standard deviation calculator, 63 Brightness / darkness criterion determiner, 64 Code designer, 71 Representative value acquisition unit, 72 Brightness / darkness determination unit, 73 Bright area encoding unit, 74 Dark area encoding unit, 75 MUX, 81 Prediction unit, 82 Difference unit, 83 Quantization unit, 84 Encoding unit, 91 Prediction unit, 92 Difference unit, 93 Quantization unit, 94 Encoding unit, 111 112 Brightness / Darkness Determination Unit, 121 Dark Area Encoding Unit, 122 Representative Value Acquisition Unit, 123 Encoder Core, 131 Comparison Unit, 141 Threshold Adjustment Unit, 151 Prediction Method Determination Unit, 152 Prediction Unit, 153 Calculation Unit, 154 Quantization Unit, 155 Inverse Quantization Unit, 156 Calculation Unit, 157 Reference Buffer, 158 Reversible Encoding Unit, 159 Stream Generation Unit, 171 Table Selection Unit, 172, 181 Entropy Encoding Unit, 201 Stream Expansion Unit, 202 Reversible Decoding Unit, 203 Inverse Quantization Unit, 204 Prediction Unit, 205 Calculation Unit, 206 Reference Buffer, 211 Table Selection Unit, 212, 221 Entropy Decoding Unit, 231 Brightness / Darkness Determination Unit, 232 Bright Area Encoder Core, 233 Dark Area Encoder Core, 234 Stream Coupling Unit, 240 Bright Area Acquisition Unit, 241 Prediction Method Determination Unit, 242 Prediction Unit, 243 Calculation Unit, 244 Quantization Unit, 245 Inverse Quantization Unit, 246 Calculation Unit, 247 Reference Buffer, 248 Reversible Encoding Unit, 249 Stream Generation Unit, 260 Dark Area Acquisition Unit, 261 Reversible Encoding Unit, 262 Stream Generation Unit, 271 Table Selection Unit, 272, 281 Entropy Encoding Unit, 291 Table Selection Unit, 292, 301 Entropy Encoding Unit, 311 Stream Splitting Unit, 312 Bright Area Decoder Core, 313 Dark Area Decoder Core, 314 Image merging unit, 321 Stream expansion unit, 322 Reversible decoding unit, 323 Inverse quantization unit, 324 Prediction unit, 325 Calculation unit,326 Reference buffer, 331 Stream expansion unit, 332 Reversible decoding unit, 341 Table selection unit, 342, 351 Entropy decoding unit, 361 Table selection unit, 362, 371 Entropy decoding unit, 401, 402 Quantization unit, 403, 411 Inverse quantization unit, 412 Quantization unit, 421 Spatial transformation quantization unit, 422, 431 Inverse quantization inverse spatial transformation unit, 441 Quantization unit, 451 Representative value acquisition unit, 452 Quantization width determination unit, 453 Quantization unit, 501 Signal processing unit, 511 Real-time playback system, 512 DB, 513 On-demand playback system, 514 Cloud AI system, 521 Decoder, 522 Development processing unit, 523 Display unit, 531 Decoder, 532 Development Processing Unit, 533 Display Unit, 541 Decoder, 542 AI Processing Unit, 611 QBI Signal Processing Unit, 612-614 DRAM, 621-623 Encoder, 631-634, 650 Decoder, 651 Addition Processing Unit, 652 Motion Estimation Unit, 653 Motion Vector Interpolation Unit, 654 Non-blurred Image Generation Unit, 655 3DNR Unit, 901 Processing Circuit, 902 ROM, 903 RAM, 904 Bus, 905 Input / Output Interface, 906 Input Unit, 907 Output Unit, 908 Storage Unit, 909 Communication Unit, 910 Drive, 911 Removable Media

Claims

The maximum number of photons is determined by counting the photons incident on the pixel during a predetermined counting period, for the number of periods in the exposure time. Based on the count value obtained from the photons, the pixel values ​​are acquired using a CLK-DET (clocked detector) method. Based on the encoding parameters generated based on the CLK-DET parameters of the CLK-DET method, the captured image composed of the pixel values ​​acquired by the CLK-DET method is encoded. encoder. 　 The encoding parameter includes a brightness threshold used to determine the brightness of the part of the captured image to be processed, Based on the brightness of the processing target portion, which is determined based on the brightness threshold, the encoding method for the processing target portion is selected. The encoder according to claim 1. 　 The brightness threshold is adjusted based on the camera parameters used when capturing the aforementioned image, and / or the signal processing parameters for the aforementioned image acquired by burst imaging. The encoder according to claim 2. 　 The aforementioned brightness threshold and, The representative value of the pixel values ​​of the processing target portion and / or the peripheral portion of the processing target portion and Based on this, the brightness of the part to be processed is determined. The encoder according to claim 2. 　 Determining the brightness and darkness of each pixel in the area to be processed. The encoder according to claim 2. 　 The coding parameters include coding information relating to the entropy code, Based on the aforementioned coding information, the portion of the captured image to be processed is entropy encoded. The encoder according to claim 1. 　 Based on the portion to be processed and / or the surrounding portion of the portion to be processed, an entropy code is selected to entropy encode the portion to be processed. The encoder according to claim 6. 　 The encoding parameters include a CLK-DET probability distribution in which photons are incident during the count period in the exposure time, Based on the CLK-DET probability distribution, the portion of the captured image to be processed is encoded. The encoder according to claim 1. 　 Based on the CLK-DET probability distribution, the quantization width for encoding the portion to be processed is set. The encoder according to claim 8. 　 The maximum number of photons is determined by counting the photons incident on a pixel during a predetermined counting period, which is performed for the number of periods in the exposure time. Based on the count value obtained from the photons, the pixel values ​​are acquired using a CLK-DET (clocked detector) method. Based on the encoding parameters generated based on the CLK-DET parameters of the CLK-DET method, the encoded data is decoded by an encoding method that encodes the captured image composed of the pixel values ​​acquired by the CLK-DET method. Decoder. 　 In the above encoding method, The encoding parameter includes a brightness threshold used to determine the brightness of the part of the captured image to be processed, Based on the brightness of the processing target portion, which is determined based on the brightness threshold, the encoding method for the processing target portion is selected. The decoder according to claim 10. 　 In the encoding method, the brightness threshold is adjusted based on the camera parameters used when capturing the captured image, and / or the signal processing parameters for the captured image acquired by burst imaging. The decoder according to claim 11. 　 In the above encoding method, The aforementioned brightness threshold and, The representative value of the pixel values ​​of the processing target portion and / or the peripheral portion of the processing target portion and Based on this, the brightness of the part to be processed is determined. The decoder according to claim 11. 　 In the above encoding method, the brightness and darkness of each pixel in the processing target portion are determined. The decoder according to claim 11. In the above encoding method, The coding parameters include coding information relating to the entropy code, Based on the aforementioned coding information, the portion of the captured image to be processed is entropy encoded. The decoder according to claim 10. 　 In the encoding method, an entropy code is selected to entropically encode the portion to be processed, based on the portion to be processed and / or the portion surrounding the portion to be processed. The decoder according to claim 15. 　 In the above encoding method, The encoding parameters include a CLK-DET probability distribution in which photons are incident during the count period in the exposure time, Based on the CLK-DET probability distribution, the portion of the captured image to be processed is encoded. The decoder according to claim 10. 　 In the encoding method, the quantization width for encoding the portion to be processed is set based on the CLK-DET probability distribution. The decoder according to claim 17. 　 Multiple pixels that perform photoelectric conversion, The maximum number of photons is determined by counting the photons incident on the pixel during a predetermined counting period, which is performed for the number of periods in the exposure time. Based on the count value obtained from the photons, the pixel values ​​are acquired using a CLK-DET (clocked detector) method. An encoder then encodes the captured image composed of the pixel values ​​acquired by the CLK-DET method, based on encoding parameters generated based on the CLK-DET parameters of the CLK-DET method. A light sensor including [this]. 　 The CLK-DET parameters are the length of one or more of the counting periods, and the number of periods for each of the counting periods. The system further includes an encoding parameter generator that generates the encoding parameters based on the CLK-DET parameters. The optical sensor according to claim 19.

Images