Image quality adjustment device, image quality adjustment method, and program

Abstract

This image quality adjustment device comprises a storage unit and an image quality conversion unit that converts the image quality of a photographed real image by using tone curves. The tone curves (TC-11 to TC-14) are curves in which, when normalization is performed with the maximum value of the luminance set to 1 and the minimum value thereof set to 0, the normalized luminance value of a converted image corresponding to the minimum value is greater, by a first offset (OS1), than the value, corresponding to said minimum value, on a straight line having a slope of 1 and an intercept of 0 when both the input and output axes are on a linear scale.

Description

Image quality adjustment device, image quality adjustment method, and program

[0001] This invention relates to an image quality adjustment device, an image quality adjustment method, and a program.

[0002] In recent years, digital cameras have seen advancements in dynamic range, enabling the capture of high-contrast, high-resolution images of subjects. These real-world images are also known as PRI (Photographed Real Image).

[0003] Japanese Patent Publication No. 2006-165826

[0004] Human vision can handle an extremely wide range of brightness. For example, on a moonlit night, a person can distinguish the patterns on the moon while also being able to discern what is in the dark areas. On the other hand, when a camera captures a moonlit scene, if you try to make bright areas like the moon clear, the dark areas will be underexposed, and if you try to make objects in the dark areas clear, the bright areas will be overexposed. Adjusting the image quality of a real-life photograph to have a wide tonal range that closely matches human perception is extremely difficult.

[0005] This invention was made under the circumstances described above, and aims to provide an image quality adjustment device, an image quality adjustment method, and a program that can adjust live-action images to an image quality with good visibility.

[0006] To achieve the above objective, the image quality adjustment device according to the first aspect of the present invention comprises: a storage unit that stores a tone curve for converting the image quality of a first image; and an image quality conversion unit that converts the image quality of the first image using the tone curve read from the storage unit, wherein the tone curve is a curve in which, when the maximum value of luminance is set to 1 and the minimum value to 0, the normalized luminance value of the converted second image corresponding to the minimum value is a value that is by a first offset greater than the value corresponding to the minimum value of a straight line with a slope of 1 and an intercept of 0 when both the input and output axes are set to a linear scale.

[0007] In the tone curve, the first offset may be set such that the contrast ratio that the displayed second image can take is between 10:1 and 200:1.

[0008] In the tone curve, the first offset may be set to decrease as the contrast ratio that the displayed second image can take increases.

[0009] The tone curve may be defined as a curve in which the normalized luminance value of the second image corresponding to the maximum value is smaller by a second offset than the value corresponding to the maximum value of a straight line with a slope of 1 and an intercept of 0, when both the input and output axes are set to a linear scale.

[0010] In the tone curve, the second offset may have a magnitude of 10% or less of the brightness range that the second image can take.

[0011] The tone curve is such that when the normalized luminance value of the first image is expressed on a log-log axis (common logarithmic scale) when the grayscale representation of the first image is 8-bit SDR (Standard Dynamic Range), it is 10 －３．５ That's 10 －１．５ The slope is greatest in a smaller range, and the normalized luminance value of the first image is 10 when expressed on a log-log axis (common logarithmic scale) if the grayscale representation of the first image is 16-bit HDR (High Dynamic Range). －4．５ That's 10 －１．５ The curve may be such that the slope is maximum in a smaller range, and the slope asymptotically approaches zero as the brightness value of the first image approaches the maximum and minimum values.

[0012] The tone curve may be such that, when both the input and output axes are on a linear scale, the slope is steep in the low-luminance region and shallow in the high-luminance region.

[0013] If the first image is an image that has been transformed such that the contrast ratio between the low-luminance region and the high-luminance region is increased when both the input and output axes are set to a linear scale, then the tone curve may be a curve with a steeper slope in the medium-luminance region than in the low-luminance and high-luminance regions.

[0014] The memory unit stores the following tone curves: a first tone curve for obtaining an image that conforms to the visual characteristics of a person viewing a landscape outdoors at night; a second tone curve for obtaining an image that conforms to the visual characteristics of a person viewing a landscape outdoors in the morning during the day; a third tone curve for obtaining an image that conforms to the visual characteristics of a person viewing a landscape outdoors in the afternoon during the day; and a fourth tone curve for obtaining an image that conforms to the visual characteristics indoors. The magnitude of the first offset may increase in the order of the second, third, fourth, and first tone curves.

[0015] The system may also include an imaging unit for capturing the first image, an output unit for outputting the second image, and an exposure time adjustment unit for adjusting the exposure time of the imaging unit according to the operation input.

[0016] The system includes an environmental condition acquisition unit that acquires ambient or imaging environmental conditions, and a tone curve adjustment unit that adjusts the tone curve used for converting the first image in the image quality conversion unit based on the environmental conditions acquired by the environmental condition acquisition unit, wherein the storage unit stores a plurality of tone curves corresponding to each of a plurality of different environmental conditions, and the tone curve adjustment unit may either select a tone curve corresponding to the environmental conditions acquired by the environmental condition acquisition unit from among the plurality of tone curves stored in the storage unit, or generate a tone curve corresponding to the environmental conditions acquired by the environmental condition acquisition unit by interpolating two of the plurality of tone curves.

[0017] The first image may be a real-world image that has been inversely gamma corrected.

[0018] The first image and the second image may both be SDR (Standard Dynamic Range) images.

[0019] The aforementioned tone curve may be a global tone mapping.

[0020] A second aspect of the present invention relates to an image quality adjustment method performed by an image quality adjustment device, which includes a conversion step of converting the image quality of a first image using a tone curve for converting the image quality of a first image, wherein the tone curve is a curve in which, when the maximum value of luminance is set to 1 and the minimum value to 0, the normalized luminance value of the converted second image corresponding to the minimum value is greater by a first offset than the value corresponding to the minimum value of a straight line with a slope of 1 and an intercept of 0 when both the input and output axes are on a linear scale.

[0021] A program according to a third aspect of the present invention causes a computer to function as a storage unit that stores a tone curve for converting the image quality of a first image, and an image quality conversion unit that converts the image quality of the first image using the tone curve read from the storage unit, wherein the tone curve is a curve in which, when the maximum value of luminance is set to 1 and the minimum value to 0, the normalized luminance value of the converted second image corresponding to the minimum value is a value that is greater by a first offset than the value corresponding to the minimum value of a straight line with a slope of 1 and an intercept of 0 when both the input and output axes are on a linear scale.

[0022] According to the present invention, live-action images can be adjusted to an image quality with good visibility.

[0023] This is a block diagram showing the functional configuration of the image quality adjustment device according to Embodiment 1 of the present invention. This is a diagram showing the details of the tone curve. This is a graph showing multiple types of tone curves. This is a diagram showing an example of a real-world image. This is a diagram showing an example of an image converted by the first tone curve. This is a diagram showing an example of an image converted by the first tone curve. This is a diagram showing an example of an image converted by a tone curve where both the first and second offsets are 0. This is a diagram showing an example of a real-world image when the exposure time is 16 times longer. This is a diagram showing an example of a real-world image. This is a diagram showing an example of an image converted by the second tone curve. This is a diagram showing an example of an image converted by the third tone curve. This is a diagram showing an example of an image converted by the fourth tone curve. This is a graph showing the output brightness characteristics of a displayed image and the reflective brightness characteristics of a printed image, taking viewing flare into consideration. This is a graph showing the output brightness characteristics of an image displayed on a display device in a dark place. This is a graph showing the output brightness characteristics of an image displayed on a display device indoors. This is a graph showing the reflective brightness characteristics of an image printed on semi-gloss paper. This is a graph showing the reflective brightness characteristics of an image printed on plain paper. This is a block diagram showing the hardware configuration of the image quality adjustment device in Figure 1. This is a flowchart of the image quality adjustment process of the image quality adjustment device in Figure 1. This is a block diagram showing the functional configuration of the image quality adjustment device according to Embodiment 2 of the present invention. Figure 12 is a flowchart of the image quality adjustment process of the image quality adjustment device. This figure shows an example of a converted image transformed by the first tone curve. This figure shows an example of a converted image transformed by the first tone curve from a real-life image with adjusted exposure time. This is a block diagram showing the functional configuration of the image quality adjustment device according to Embodiment 3 of the present invention. Figure 15 is a flowchart of the image quality adjustment process of the image quality adjustment device. This is a graph showing the tone curve corresponding to how the uncorrected image appears on the display device. This is a graph showing the tone curve corresponding to how the corrected image appears on the display device. This figure shows the fifth tone curve for LDR display. This figure shows the relationship between 8-bit sRGB (standard RGB) values, normalized values, and contrast ratio. This figure shows a modified example of the tone curve. This figure shows a comparison of the image quality conversion by the tone curve in Figure 20 and the histograms of the images before and after conversion. This figure shows a first example of image quality conversion by the tone curve in Figure 20.This figure shows a second example of image quality conversion using the tone curve in Figure 20. This figure shows a third example of image quality conversion using the tone curve in Figure 20. This figure shows the conversion result of an 8-bit real-world image using the tone curve in Figure 20, and the conversion result of a 16-bit real-world image using the tone curve in Figure 20. This figure shows a fourth example of image quality conversion using the tone curve in Figure 20. This figure shows an example of a tone curve at night. This figure shows an example of a tone curve during the day. This is an 8-bit SDR image taken during the day with a relative exposure of + / - 0 EV (Exposure Value). This is a 16-bit HDR image taken during the day with a relative exposure of -5.3 EV. This is an image obtained by applying a tone curve to the image in Figure 28B. This is an 8-bit image obtained by applying local processing to the image in Figure 28B. This is an image taken with a smartphone. This figure shows the size of the first offset in the corresponding gradation. This is the first figure showing the image converted using the tone curve in Figure 27A. This is the second figure showing the image converted using the tone curve in Figure 27A. This is the third figure showing the image converted using the tone curve in Figure 27A. Figure 1 shows the image transformed using the tone curve in Figure 27B. Figure 2 shows the image transformed using the tone curve in Figure 27B. Figure 3 shows the image transformed using the tone curve in Figure 27B.

[0024] Embodiments of the present invention will now be described in detail with reference to the drawings. In all drawings, identical or corresponding components are denoted by the same reference numerals.

[0025] Embodiment 1 First, Embodiment 1 of the present invention will be described. The image quality adjustment device 1A according to this embodiment shown in Figure 1 converts the image quality of a real-life image P1 as a first image. For example, the image quality adjustment device 1A converts the image quality of a real-life image P1 captured by the imaging device 30.

[0026] The imaging device 30 obtains a real-world image P1 by imaging. The real-world image P1 is an image captured with a predetermined dynamic range, for example, SDR (Standard Dynamic Range) or HDR (High Dynamic Range). Dynamic range refers to the range (ratio) of the maximum and minimum brightness values ​​in an image. In an SDR image, the dynamic range is, for example, 100:1 (1 / 100), and its brightness value is represented by 8 bits for gradation. In an HDR image, the dynamic range is, for example, 6000:1 (1 / 6000), and its brightness value is represented by more than 8 bits (for example, 16 bits) for gradation.

[0027] The actual image P1 is, for example, an image captured by automatic exposure in accordance with the characteristics of the image sensor of the imaging device 30, which has been inversely gamma corrected to make its characteristics linear. In this case, the actual image P1 may not be a simple output of the image sensor, but an image to which irreversible image quality enhancement processing, in which global processing and local processing are additionally performed, has been applied. In this case, inverse gamma correction may be performed by the nominal gamma value assigned to the actual image P1. Furthermore, if the actual image P1 is an SDR image rather than an HDR image, a pre-processing step may be added in which the 8-bit image is first converted to a 16-bit image (pseudo-HDR image) in terms of grayscale resolution in order to reduce the expansion of quantization errors due to internal processing. The image quality adjustment device 1A converts this actual image P1 into a converted image P2, which is a second image with good visibility and in accordance with human visual characteristics. The converted image P2 is a Standard Representational Image (SRI). Standard representation images have a brightness that closely resembles how a person would perceive a subject when viewing it directly, within a grayscale range equivalent to SDR (e.g., 8 bits).

[0028] In order to obtain the converted image P2, the image quality adjustment device 1A includes a storage unit 10 and a conversion unit 20. The storage unit 10 stores a tone curve TC for converting the image quality of the real-shot image P1. The conversion unit 20 includes an image quality conversion unit 21 that converts the image quality of the real-shot image P1 using the tone curve TC read from the storage unit 10 and generates the converted image P2. As shown in FIG. 1, the tone curve TC is a tone curve of global tone mapping capable of tone mapping in all luminance ranges that the pixels of the real-shot image P1 can take.

[0029] The tone curve TC will be described using a coordinate system in which the horizontal axis represents the luminance value of the pixel of the real-shot image P1 before conversion, the vertical axis represents the luminance value of the pixel of the converted image P2, and both axes are logarithmically displayed. The tone curve TC is normalized with the maximum value of luminance being 1. The tone curve TC is a curve that monotonically increases as the input value approaches 1. In FIG. 2, a straight line L with a slope γ = 1 of the luminance value (output value) of the converted image P2 with respect to the luminance value (input value) of the real-shot image P1 is shown by a dashed line. The tone curve TC is above this straight line L and becomes a curve that is convex upward in the range where the input value is 0.0003 (10 －３．５ ).

[0030] The inventor has performed a number of analysis operations comparing the appearance of the real environment with the images taken in the photograph. As a result, it has been clarified that the most sensitive part of human visual characteristics is the range where the luminance range of the image is 0.0003 (10 －３．５ ) or more and 0.03 (10 －１．５ ) or less. Therefore, in the tone curve TC, as shown in FIG. 2, the slope γ is defined to be maximum in the range where the luminance value (input value) of the real-shot image P1 is 0.0003 (10 －３．５ ) or more and 0.03 (10 －１．５ ) or less. That is, in the tone curve TC, many gradations are allocated to the range where human sensitivity is high. Further, the tone curve TC is defined to be a curve whose slope asymptotically approaches 0 as the luminance value of the real-shot image P1 approaches the maximum value (1 in FIG. 1) and the minimum value (10 －３．５ ) in FIG. 1).

[0031] Generally, in the environment in which a real-world image P1 is captured, bright and dark areas appear due to the sun, moon, and illumination sources outdoors, and due to illumination sources indoors. Humans can perceive a wide range from bright to dark areas, but what humans primarily see is not the bright areas such as the sun, moon, and illumination sources, but mainly the luminance range darker than those, which is 0.0003 (10 －３．５ ) or more 0.03 (10 －１．５ ) The range is as follows:

[0032] Furthermore, the tone curve TC is a curve in which the luminance value of the converted image P2 corresponding to the minimum luminance value of the actual image P1 is greater by the first offset OS1 than the value corresponding to the same minimum value of the straight line L with a slope γ of 1 and an intercept of 0. Note that, as shown in Figure 2, the minimum luminance value of the actual image P1 is 10 in the case of an SDR image. －３．５ This is the result. Also, in the case of HDR images, as shown in Figure 8A below, the minimum brightness value of the actual image P1 is 10 －３．５ Because it becomes smaller, the effective dynamic range increases by about 10 times, －４．５ This is how it works. Regardless of the minimum value of the luminance of the real-world image P1, a first offset OS1 is set at that minimum value in the tone curve TC. By defining the first offset OS1, if the luminance value of a pixel in the real-world image P1 is 0, the luminance value of that pixel is converted to OS1. Even in areas that humans perceive as dark and completely like a "hole of darkness," it is possible to perceive some kind of image in that area, rather than it being black as a surface color. By setting the first offset OS1 and shifting the luminance value of the image representing the darkest part slightly to the higher side, the image in the dark parts can be made easier to see in the converted image P2.

[0033] The signal emitted from the human photoreceptor cells that receive light is an electrical signal, and noise components are mixed into this electrical signal. For example, the quantization noise of light generated in photoreceptor cells is a typical one of such noise components. Such noise components have a great impact on the visual characteristics of humans in the dark region. On the other hand, the electrical signal emitted from the imaging element of the imaging device 30 also contains noise components. Such noise components include, for example, the thermal noise of the imaging element. In terms of including noise components, the imaging device 30 has the same properties as human photoreceptor cells. By providing the first offset OS1 to the tone curve TC and shifting the luminance value of the image representing the darkest part slightly higher, the noise components due to the thermal noise of the imaging element can be emphasized, and a converted image reflecting the noise components that affect human visual characteristics can be generated. In the present embodiment, mainly, the case where the actual photographed image P1 and the converted image P2 are only SDR images, rather than those obtained by expanding the gradation luminance range of the image to HDR, will be described.

[0034] In the tone curve TC, it is desirable that the first offset OS1 be 10% or less of the range of the luminance values of each pixel that the converted image P2 can take. This is because if the first offset OS1 is made larger than this, the luminance range that the converted image P2 can take becomes too small.

[0035] Furthermore, the tone curve TC is a curve such that the luminance value of the converted image P2 corresponding to the maximum value of the luminance values of the actual photographed image P1 is a value smaller by the second offset OS2 than the value corresponding to the maximum value of the straight line L with a slope γ of 1 and an intercept of 0. By defining the second offset OS2, the luminance value 1 of the pixel of the actual photographed image P1 is converted to the luminance value 1 - OS2. Even for a part that is felt to be dazzling and cannot be stared at directly, a human can perceive an image in that part. By providing the second offset OS2 and shifting the luminance value of the image representing the brightest part slightly lower, the image of the bright part can be made easier to see.

[0036] In tone curve TC, it is desirable that the second offset OS2 be 10% or less of the luminance range that the converted image P2 can take. If the second offset OS2 is made larger than this, the range that the converted image P2 can take will become too small.

[0037] As shown in Figure 3, the memory unit 10 actually stores tone curves TC-1, TC-2, TC-3, and TC-4 as tone curves TC. Tone curves TC-1, TC-2, TC-3, and TC-4 are defined in accordance with the brightness that a person would perceive when directly viewing the subject in the actual image P1, in an environment related to the visual characteristics of the person in which the actual image P1 was taken (time of day, whether outdoors or indoors). Tone curve TC-1, as the first tone curve, is a tone curve for obtaining an image that conforms to the visual characteristics of a person viewing a landscape outdoors at night. Tone curve TC-2, as the second tone curve, is a tone curve used to convert an actual image P1 taken outdoors in the morning during the daytime. Tone curve TC-3, as the third tone curve, is a tone curve used to convert an actual image P1 taken outdoors in the afternoon during the daytime. Tone curve TC-4, as the fourth tone curve, is a tone curve for obtaining an image that conforms to the visual characteristics indoors. Through conversion using tone curves TC-1 to TC-4, the actual image P1 becomes an image that conforms to the visual characteristics of a person viewing the landscape in each environment. Tone curves TC-1 to TC-4 are curves obtained by showing images to multiple people and having them select what kind of environment the image was taken in, and then statistically analyzing the results of their selections. In other words, although there are individual differences in how things are seen, tone curves TC-1 to TC-4 are curves that show the average value of how people see things, statistically calculated from multiple samples according to the set environmental conditions.

[0038] In the graph shown in Figure 3, the brightness value of the real-world image P1 is on the horizontal axis, and the brightness value of the converted image P2 is on the vertical axis. In Figure 3, the horizontal axis is 10 －７ The brightness range is taken from 1, and the vertical axis is 10 －３ It takes a brightness range of 1 from there.

[0039] When the tone curves TC-1 to TC-4 are normalized with the maximum brightness value set to 1 and the minimum value to 0, the brightness value of the actual image P1 is 10 －３．５ That's 10 －１．５ These curves share the characteristic that the slope γ is maximum in a smaller range, and the slope asymptotically approaches 0 as the brightness value of the real-world image P1 approaches its maximum and minimum values.

[0040] Furthermore, the first offset OS1 increases in the order of tone curves TC-2, TC-3, TC-4, and TC-1. This is thought to be because, in brighter environments, even if the first offset OS1 is reduced, the importance of the darkest areas decreases relatively in terms of human visual characteristics. In tone curve TC-1, the first offset OS1 is 0.05 (5% of the luminance range). In darker environments, raising the low-frequency curve makes it easier to bring out images in the dark areas.

[0041] The second offset OS2 is almost the same for tone curves TC-1 to TC-4. For tone curves TC-1 to TC-4, the second offset OS2 is 0.10 (10% of the luminance range) or less.

[0042] The image quality conversion unit 21 converts the image quality of the actual image P1 using tone curves TC-1 to TC-4. The image quality conversion unit 21 converts the image quality of the actual image P1 using the tone curve TC from among tone curves TC-1 to TC-4 that corresponds to the environment in which the image was taken. For example, if the actual image P1 was taken outdoors at night, tone curve TC-1 is used; if the actual image P1 was taken outdoors in the morning, tone curve TC-2 is used; and if the actual image P1 was taken outdoors in the afternoon, tone curve TC-3 is used. Also, for example, if the actual image P1 was taken indoors, tone curve TC-4 is used. Tone curves TC-1 to TC-4 are registered in a Look Up Table (LUT). By referring to the LUT, the output value for the input value can be obtained directly.

[0043] Figure 4A shows an actual SDR image P1 taken at night. Figure 4B shows a converted image P2, in which the image quality has been transformed using the tone curve TC-1, which is suitable for nighttime. Compared to the actual image P1 in Figure 4A, the converted image P2 in Figure 4B is brighter overall, improving its visibility and making it closer to how the landscape would actually appear to a person.

[0044] Figure 5A shows a magnified portion of the converted image P2 using tone curve TC-1. On the other hand, Figure 5B shows a magnified portion of the converted image P2' using a tone curve with the same shape as tone curve TC-1, but with the first offset OS1 and second offset OS2 set to 0. Furthermore, as shown in Figure 5C, an HDR image (actual image P1') captured with an exposure time 16 times longer is shown. Comparing Figure 5A and Figure 5B, it can be seen that the converted image P2 in Figure 5A has fewer areas of black crushing compared to Figure 5B. The percentage of black crushing in the actual image P1' in Figure 5C is almost the same as the percentage of black crushing in the converted image P2' in Figure 5B. Thus, the converted image P2 has a brightness similar to the HDR image (actual image P1') captured with an exposure time 16 times longer, and the contrast in dark areas is greater than that of the HDR image (converted image P2').

[0045] Furthermore, Figure 6A shows the actual image P1, Figure 6B shows the converted image P2 obtained by converting the actual image P1 in Figure 6A using the tone curve TC-2 corresponding to the morning, Figure 6C shows the converted image P2 obtained by converting the actual image P1 in Figure 6A using the tone curve TC-3 corresponding to the afternoon, and Figure 6D shows the converted image P2 obtained by converting the actual image P1 in Figure 6A using the tone curve TC-4 corresponding to indoors. The converted image P2 in Figure 6B is closer to how a person would actually see the subject in the morning than the actual image P1 in Figure 6A, and the converted image P2 in Figure 6C is closer to how a person would actually see the subject in the afternoon. The image in the dark areas is clearer in the converted image P2 in Figure 6B than in the converted image P2 in Figure 6C. The real-world image P1 shown in Figure 6A is an outdoor image. However, by using the tone curve TC-4, which corresponds to indoors, to transform this real-world image P1, the transformed image P2 can be made to closely resemble how it would look indoors.

[0046] [Display Device] Returning to Figure 1, the display device 40 is a light-emitting display that displays the converted image P2. The display device 40 performs corrections on the converted image P2 according to the characteristics of its own hardware, and then displays the converted image P2. There are various types of display devices 40, for example, there are SDR-compatible ones and HDR-compatible ones.

[0047] [Printing device] The printing device 50 prints the converted image P2 onto the paper. The printing device 50 applies corrections to the converted image P2 according to the characteristics of its own hardware before printing the converted image P2. Examples of paper used for printing include plain paper and high-quality semi-gloss paper.

[0048] [Actual Output Brightness Characteristics] When viewing the screen of the display device 40 in a brightly lit room, external illumination light enters the screen, and the reflected light reduces the contrast of the image displayed on the screen. This noise that reduces contrast is called viewing flare (black floating). On the other hand, the paper printed by the printing device 50 originally requires illumination light to view the image on the paper, but the maximum density of the print (black) corresponding to an input brightness of 0 does not become infinite, but remains at a constant value. Therefore, the reflected light reduces the contrast of the image displayed on the paper. In this case, strictly speaking, the additive model of internal light emission and viewing flare in an emissive display like the display device 40 cannot be used, and a more complex model is required. However, it is possible to treat them in a similar manner for approximation. Therefore, for the sake of simplicity in the following explanation, we will refer to them collectively as viewing flare.

[0049] The tone curves TC-1 to TC-4 described above take into account viewing flare when the converted image P2 is displayed on the display device 40. In other words, even when the converted image P2 is displayed on a display screen that reflects very bright ambient light, it is possible to obtain a considerable amount of information with a contrast close to reality, and furthermore, it is possible to obtain information about the dark areas of the room. Viewing flare is disclosed in detail, for example, in International Publication WO2018 / 212212.

[0050] Figure 7 shows the output luminance characteristics and reflective luminance characteristics, taking into account viewing flare, when viewing the screen of the display device 40 showing the converted image P2 converted using the tone curve TC-1 and the printed material from the printing device 50. First, when the converted image P2 displayed on the screen of the display device 40 is viewed in a dark place (equivalent to CR = 1000:1), the contrast is high because there is little viewing flare. Therefore, in a dark place, the converted image P2 can be viewed in a clear state.

[0051] On the other hand, when the screen of the display device 40 is viewed in a brightly lit room with sunlight (but without direct illumination of the screen) (equivalent to CR = 100:1), viewing flare increases, resulting in lower contrast than in a dark environment. However, because the slope of the output brightness characteristic is not zero across the entire input range, the converted image P2 can be viewed without image distortion.

[0052] Figure 7 also shows the reflective brightness characteristics when the printed image P2 is printed on semi-gloss paper using the printing device 50 and viewed in a room with sunlight, and when the printed image is printed on plain paper and viewed in a room. The contrast is lower on plain paper than on semi-gloss paper. However, whether it is plain paper or semi-gloss paper, the slope of the characteristics is not zero across the entire input range, so the printed converted image P2 can be viewed without image crushing.

[0053] Figure 8A shows the output brightness characteristics, taking into account viewing flare, when viewing the screen of a display device 40 displaying a converted image P2 (see, for example, Figure 4B) converted using various tone curves in a dark environment (contrast ratio: equivalent to 1000:1). The curve indicated by connecting the squares shows the output brightness characteristics of the converted image P2 using tone curve TC-1. The curve indicated by connecting the triangles shows the output brightness characteristics of a converted image using a tone curve that is almost the same shape as tone curve TC-1, with the first offset OS1 and second offset OS2 being 0. The curve indicated by connecting the crosses shows the output brightness characteristics of a converted image that has been converted twice using tone curve TC-1. The curve indicated by connecting the dots shows the output brightness characteristics of a normal SDR image with an exposure time 16 times longer. As shown in Figure 8A, in the low frequency range, tone curve TC-1 (square) is shifted upwards compared to the tone curves (▲ and ●), and has a shape that converts the image in dark areas to be brighter. Note that the input luminance range W shown in Figure 8A is the expanded region when the input is expanded from an SDR image to an HDR image. Therefore, the difference in output luminance characteristics within the input luminance range W becomes more pronounced when the input image is an HDR image.

[0054] Figure 8B shows the luminance output characteristics, taking viewing flare into account, when viewing the screen of a display device 40 showing converted images P2 converted using various tone curves in an indoor environment (contrast ratio: equivalent to 100:1). As shown in Figure 8B, in the low-frequency range of the input, the output luminance characteristics of tone curve TC-1, indicated by the squares, are shifted upwards compared to the curves ▲ and ●. This indicates that the converted image P2 using tone curve TC-1 has better visibility in the low-frequency range than images converted with other tone curves. The range W is the same as the range W in Figure 8A.

[0055] Figure 9A shows a comparison of reflective luminance characteristics considering viewing flare when viewing semi-gloss paper (contrast ratio: equivalent to 20:1) printed with converted images converted using various tone curves, and Figure 9B shows a comparison of reflective luminance characteristics considering viewing flare when viewing plain paper (contrast ratio: equivalent to 10:1) printed with converted images converted using various tone curves. As shown in Figures 9A and 9B, there are no significant differences in the output luminance characteristics of the three tone curves, except for the curve marked with ×.

[0056] [Hardware Configuration] Figure 10 shows the hardware configuration of the image quality adjustment device 1A. As shown in Figure 10, the image quality adjustment device 1A includes a CPU (Central Processing Unit) 60, a memory 61, an external storage unit 62, an operation unit 63, a display unit 64, a communication interface (I / F) 65, and an input / output unit 66. Each component of the image quality adjustment device 1A is connected via an internal bus 70.

[0057] The CPU 60 is a processor (arithmetic unit) that executes software programs (hereinafter simply referred to as "programs"). The program 69 is read into the memory 61 from the external storage unit 62, and the CPU 60 performs conversion processing using the tone curve TC by executing the program 69 stored in the memory 61. In other words, the image quality conversion unit 21 operates as a result of the CPU 60 executing the program 69.

[0058] Memory 61 is, for example, RAM (Random Access Memory). Memory 61 stores the program 69 executed by the CPU 60, as well as data necessary for the CPU 60 to execute program 69 (tone curve TC, actual image P1, converted image P2, etc.), and data generated as a result of the execution of program 69.

[0059] The external storage unit 62 is, for example, a hard disk. The external storage unit 62 stores the program 69 executed by the CPU 60. In addition, the program 69 is stored on a recording medium 80 such as a portable USB (Universal Serial Bus) memory. The external storage unit 62 stores the program 69 transferred from the recording medium 80.

[0060] The control unit 63 is a man-machine interface that can be operated by an operator. Examples of the control unit 63 include a keyboard, mouse, touch panel, camera, microphone, etc. The CPU 60 operates according to the operation input from the control unit 63. The operation input to specify whether or not to convert the actual image P1 using one of the tone curves TC-1 to TC-4 is performed by the control unit 63.

[0061] The display unit 64 is a display that shows images. The display unit 64 outputs an image signal output from the CPU 60. As a result, the display unit 64 displays an image based on that image signal. In the case of a touch panel, the operation unit 63 and the display unit 64 are integrated. The display unit 64 can display a real-life image P1 and a converted image P2. It is also possible to use the display unit 64 in place of the display device 40.

[0062] The communication interface 65 is a communication interface for communicating with external devices. The communication interface 65 is an input / output interface for the communication network. The image quality adjustment device 1A is connected to the imaging device 30, the display device 40, and the printing device 50 via this communication interface 65, and inputs the actual image P1 and outputs the converted image P2 to the outside.

[0063] The input / output unit 66 is the input / output interface for the recording medium 80. The program 69 or the actual image P1 is input via this input / output unit 66. The converted image P2 can also be output to the recording medium 80 via this input / output unit 66.

[0064] [Operation of the Image Quality Adjustment Device] Next, the image quality adjustment method (image quality adjustment process) performed by the image quality adjustment device 1A according to this embodiment will be described. Figure 11 shows a flowchart of the image quality adjustment process of the image quality adjustment device 1A.

[0065] As shown in Figure 11, first, the image quality conversion unit 21 receives an actual image P1 from the imaging device 30 or the like (step S1). Furthermore, the image quality conversion unit 21 sets environmental conditions based on the operation input (step S2). Here, for example, nighttime, morning, afternoon, indoors, etc., are specified as environmental conditions by the operation input. Subsequently, the image quality conversion unit 21 selects a tone curve TC according to the environmental conditions specified by the operation input and reads it from the storage unit 10 (step S3). For example, if nighttime is specified by the operation input, the image quality conversion unit 21 selects tone curve TC-1 and reads it from the storage unit 10. If morning is specified by the operation input, the image quality conversion unit 21 selects tone curve TC-2 and reads it from the storage unit 10. If afternoon is specified by the operation input, the image quality conversion unit 21 selects tone curve TC-3 and reads it from the storage unit 10. If indoors is specified by the operation input, the image quality conversion unit 21 selects tone curve TC-4 and reads it from the storage unit 10.

[0066] Next, the image quality conversion unit 21 converts the image quality of the live-action image P1 using the read tone curve TC (step S4). This step is a conversion step in which the image quality of the live-action image P1 is converted using the tone curve TC of global tone mapping, which converts the image quality of the live-action image P1 to an image quality that conforms to the characteristics of human vision. When the tone curve TC is normalized with the maximum brightness value set to 1 and the minimum value set to 0, the brightness value of the live-action image P1 is 10 －３．５ That's 10 －１．５As previously mentioned (see Figure 2), the curve has a maximum slope over a smaller range, and as the brightness value of the real-world image P1 approaches its maximum and minimum values, the slope asymptotically approaches 0. The brightness value of the converted image P2 corresponding to the minimum value is greater by a first offset OS1 than the value corresponding to the minimum value on a straight line L with a slope of 1 and an intercept of 0, and the brightness value of the converted image P2 corresponding to the maximum value is smaller by a second offset OS2 than the value corresponding to the maximum value on a straight line L with a slope of 1 and an intercept of 0. This conversion step allows for adjustment of image quality with a wide gradation characteristic close to human perception, resulting in good visibility.

[0067] Finally, the image quality conversion unit 21 outputs the converted image P2, whose image quality has been converted, to the display device 40 or the printing device 50 (step S5). The converted image P2 is output to the display device 40 and displayed on its screen, or output to the printing device 50 and printed. As a result, the converted image P2 is displayed or printed in accordance with human visual characteristics.

[0068] The tone curve TC is essentially a curve that narrows the brightness range of the converted image P2, and it might seem that image quality conversion using the tone curve TC would actually decrease contrast. However, this tone curve TC improves image reproduction across a wide brightness range by shifting the dark areas towards brighter areas and the bright areas towards darker areas, making images present in both dark and bright areas easier to see. As explained in Figure 5, even with SDR images, good tonal reproduction can be obtained by using the tone curve TC.

[0069] The conversion using tone curve TC results in an increase in sensitivity of approximately 16 times (Δ = +4.0 EV) at night, and an increase of 8 times (Δ = +3.0 EV) to 16 times (Δ = +4.0 EV) during the day. This makes it possible to obtain brighter images with clearer details in dark areas, even with SDR images.

[0070] Embodiment 2 Next, Embodiment 2 of the present invention will be described. The image quality adjustment device 1B according to this embodiment differs from the image quality adjustment device 1A according to Embodiment 1 in that it not only adjusts the image quality of the actual image P1 but also adjusts the exposure time of the imaging device 30.

[0071] As shown in Figure 12, the image quality adjustment device 1B is the same as the image quality adjustment device 1A in that it includes a storage unit 10 and a conversion unit 20. Furthermore, the image quality adjustment device 1B includes an imaging device 30 that captures a real-life image P1 as an imaging unit, a display device 40 that displays the converted image P2, and a printing device 50 that prints the converted image P2. The display device 40 and the printing device 50 correspond to the output unit.

[0072] The conversion unit 20 includes an exposure time adjustment unit 22 in addition to the image quality conversion unit 21. The image quality conversion unit 21 converts the actual image P1 into a converted image P2, similar to the first embodiment described above. The display device 40, acting as a display unit, displays the converted image P2. The user looks at the converted image P2 displayed on the display device 40 and determines whether or not exposure time adjustment is necessary. This determination is usually made based on the purpose of use of the image, for example, whether or not the image of the area of ​​interest is visible. The exposure time adjustment unit 22 adjusts the exposure time of imaging in the imaging device 30 according to the user's operation input using the operation unit 63 shown in Figure 10. With these configurations, the user can obtain an image of the desired brightness while checking the converted image P2 and adjusting the exposure time of the imaging device 30.

[0073] Next, the image quality adjustment method, or image quality adjustment process, performed by the image quality adjustment device 1B according to this embodiment will be described. As shown in Figure 13, the input of a real-life image P1 (step S1), setting of environmental conditions (step S2), selection of the tone curve TC (step S3), image quality conversion of the real-life image P1 (step S4; conversion step), and output of the converted image P2 to the display device 40 or printing device 50 (step S5) are performed in the same order as the image quality adjustment device 1A according to Embodiment 1 described above. In step S5, the output converted image P2 is displayed on the display device 40 or printed by the printing device 50.

[0074] Furthermore, in the image quality adjustment device 1B, after step S5 is executed, the exposure time adjustment unit 22 determines whether or not to adjust the exposure time (step S6). This determination is made by referring to the converted image P2 displayed on the display device 40, as described above. The user looks at the converted image P2 displayed on the display device 40 or printed on it and decides whether or not to adjust the exposure time based on whether or not an image can be confirmed in the area of ​​interest. If it is decided to adjust the exposure time, the user inputs a command to adjust the exposure time to the image quality adjustment device 1B by operating the operation input of the operation unit 63. This command includes the amount of exposure time to be adjusted.

[0075] If the exposure time needs to be adjusted (step S6; Yes), the exposure time adjustment unit 22 changes the exposure time and outputs a command to adjust the exposure time (exposure time setting information) to the imaging device 30 (step S7). The imaging device 30 adjusts the exposure time according to the exposure time setting information and takes a new image, obtains a real image P1 taken with the adjusted exposure time, and outputs it to the image quality adjustment device 1B.

[0076] Subsequently, the image quality conversion unit 21 performs the following steps in order: input of the actual image P1 (step S1), setting of environmental conditions (step S2), selection of the tone curve TC (step S3), image quality conversion of the actual image P1 (step S4), and output of the converted image P2 (step S5). Next, the exposure time adjustment unit 22 determines whether or not to adjust the exposure time (step S6), and if adjustment is to be made (step S6; Yes), changes the exposure time (step S7). As long as it is determined that the exposure time should be adjusted (step S6; Yes), steps S1 to S7 are repeated.

[0077] If it is determined that no adjustment of the exposure time is necessary (step S6; No), the image quality adjustment device 1B terminates the image quality adjustment process.

[0078] Figure 14A shows the converted image P2 captured with a standard exposure time. Figure 14B shows the converted image P2 based on the actual image P1 captured with a longer exposure time by the image quality adjustment device 1B. As can be seen by comparing Figure 14A and Figure 14B, the converted image P2 shown in Figure 14B is overexposed. However, the image of parts that were too dark to see in the converted image P2 shown in Figure 14A is clearly visible in the converted image P2 shown in Figure 14B. The image quality adjustment device 1B can generate such a converted image P2.

[0079] The image quality adjustment device 1B according to this embodiment is suitable for, for example, nighttime surveillance cameras. This is because it can increase the exposure time of the surveillance camera and generate a converted image P2 (converted image P2 as shown in Figure 14B) in which images of dark areas can also be seen. Furthermore, the image shown in Figure 14B is an image that gives the viewer a sense of surrealism that transcends reality.

[0080] Embodiment 3 Embodiment 3 of the present invention will now be described. The image quality adjustment device 1C according to this embodiment shown in Figure 15 differs from the image quality adjustment devices 1A and 1B according to Embodiments 1 and 2 above in that it automatically acquires environmental conditions and converts the actual image P1 with a tone curve TC corresponding to the acquired environmental conditions.

[0081] As shown in Figure 15, the conversion unit 20 includes, in addition to the image quality conversion unit 21, an environmental condition acquisition unit 23 and a tone curve adjustment unit 24.

[0082] The environmental condition acquisition unit 23 acquires the surrounding environmental conditions. These environmental conditions include information such as ambient brightness, time of day, and time of day. The conversion unit 20 is connected to an environmental sensor 90 that detects the surrounding environmental conditions. For example, the environmental condition acquisition unit 23 acquires the environmental conditions detected by the environmental sensor 90. The environmental sensor 90 detects information regarding ambient brightness, time of day, and surrounding conditions.

[0083] The tone curve adjustment unit 24 adjusts the tone curve TC used for converting the actual image P1 in the image quality conversion unit 21, based on the environmental conditions acquired by the environmental condition acquisition unit 23.

[0084] The memory unit 10 stores multiple tone curves TC corresponding to multiple different environmental conditions. The tone curve adjustment unit 24 selects a tone curve TC from among the multiple tone curves TC stored in the memory unit 10 that corresponds to the environmental conditions acquired by the environmental condition acquisition unit 23. Such tone curves TC include, for example, tone curves TC-1 to TC-4 (see Figure 3), but are not limited to these. For example, multiple tone curves TC corresponding to seasons and weather conditions may be stored.

[0085] The tone curve adjustment unit 24 may generate a new tone curve TC corresponding to the environmental conditions acquired by the environmental condition acquisition unit 23 by interpolating two tone curves TC from among a plurality of tone curves TC. For example, if the time is noon, the tone curve adjustment unit 24 may generate a tone curve TC corresponding to noon by interpolating tone curve TC-2 corresponding to the morning and tone curve TC-3 corresponding to the afternoon, and use it for the conversion of the actual image P1. Alternatively, the tone curve adjustment unit 24 may generate an evening tone curve TC by interpolating the afternoon tone curve TC-3 and tone curve TC-1 corresponding to the night, and use it for the conversion of the actual image P1.

[0086] [Operation of the Image Quality Adjustment Device] Next, the image quality adjustment method (image quality adjustment process) performed by the image quality adjustment device 1C according to this embodiment will be described. Figure 16 shows a flowchart of the image quality adjustment process of the image quality adjustment device 1C.

[0087] As shown in Figure 16, first, the image quality conversion unit 21 receives a real-world image P1 from the imaging device 30 or the like (step S1). Next, the environmental condition acquisition unit 23 acquires the environmental conditions (step S11). Then, the tone curve adjustment unit 24 adjusts the tone curve according to the acquired environmental conditions (step S12). The tone curve adjustment unit 24 selects a tone curve TC from among a plurality of tone curves TC stored in the storage unit 10 that corresponds to the environmental conditions acquired by the environmental condition acquisition unit 23.

[0088] In step S12, the tone curve adjustment unit 24 may adjust the tone curve TC by interpolating two of the multiple tone curves TC to generate a tone curve TC corresponding to the environmental conditions acquired by the environmental condition acquisition unit 23, as described above.

[0089] Next, the image quality conversion unit 21 converts the image quality of the live-action image P1 using the adjusted tone curve TC (step S4; conversion step).

[0090] Finally, the image quality conversion unit 21 outputs the converted image P2, whose image quality has been converted, to the display device 40 or the printing device 50 (step S5). The converted image P2 is output to the display device 40 and displayed on its screen, or it is output to the printing device 50 and printed. This provides a converted image P2 that conforms to the characteristics of human vision. In this way, the image quality can be converted using a tone curve TC that corresponds to the environmental conditions, so that image quality suitable for the surrounding environmental conditions can be obtained.

[0091] Up to this point, we have described the case of converting the image quality of a single image, i.e., a still image IM (see Figure 15). However, the object to convert the image quality may also be a video DVD (see Figure 15). In this case as well, the processes shown in Figures 11, 13, and 16 should be executed sequentially for each frame image of the video DVD or for a predetermined number of frames.

[0092] Furthermore, the environmental conditions are not limited to those detected by the environmental sensor 90. The surrounding environmental conditions when the still image IM and video DVD are captured can be stored in some form of information, and the environmental condition acquisition unit 23 can read out the stored environmental conditions. For example, if the environmental conditions are stored in the metadata of the still image IM and video DVD, the environmental condition acquisition unit 23 can acquire the environmental conditions by reading out that metadata. In other words, the environmental conditions can be the surrounding environmental conditions when viewing the image, or the environmental conditions at the time the image was captured. In this way, the image can be viewed as if the viewer were actually there at the time of capture.

[0093] In this embodiment, the live-action image P1 and the converted image P2 may be SDR images or HDR images. Conversion from one SDR image to another is possible, and conversion from one HDR image to another is possible. In Figure 2, the position of the first offset OS1 changes in the horizontal axis direction depending on the type of converted image P2. Furthermore, conversion from an HDR image to an SDR image is also possible. With the image quality adjustment devices 1A, 1B, and 1C, even when converting from an HDR image to an SDR image, a decrease in image quality is suppressed.

[0094] There are various types of display devices 40 that display images, with contrast ratios ranging from 100:1 (CR100) to 25:1 (CR25), 10:1 (CR10), and even lower. Display devices with a contrast ratio of CR25 or lower (for example, projectors) are also called low dynamic range (LDR) display devices. Even with these LDR display devices 40, there is a need to enable users to view images with the highest possible visual contrast.

[0095] In this embodiment, viewing flare due to reflection of illumination light on the screen is assumed in advance, and image conversion is performed so that the contrast is as good as possible within the contrast ratio range of the LDR image. For example, Figure 17A shows the tone curve TC'(G ＭＡＸ In addition to the above, the converted image P2 transformed by this tone curve TC' is G ＭＩＮ (Contrast ratio less than 10:1), curves showing the actual appearance when displayed on CR25 and CR100 display devices are also shown. As shown in Figure 17A, due to viewing flare, the input value is 10 －３ In the vicinity of this point, the slope of these curves becomes smaller. Therefore, there is a risk of black clipping occurring in the image.

[0096] The memory unit 10 stores the tone curve TC-5 as the fifth tone curve. As shown in Figure 18, the tone curve TC-5 assumes that the viewing flare is 10% of the contrast ratio, and the input value 10 －３ The above 10 －２The slope γ is maximum within the following range, and as the brightness value of the input image approaches the maximum and minimum values, the curve becomes such that the slope γ asymptotically approaches 0. The brightness value of the output image corresponding to the minimum value is greater by a first offset OS1 than the value corresponding to the minimum value of a straight line L with a slope of 1 and an intercept of 0, and the brightness value of the converted image P2 corresponding to the maximum value is less by a second offset OS2 than the value corresponding to the maximum value of a straight line L with a slope of 1 and an intercept of 0.

[0097] The image quality conversion unit 21 further corrects the image quality of the converted image P2 as the first image using the tone curve TC-5 read from the storage unit 10, and generates a low-contrast corrected image as the second image. When the image quality of the image is converted using the tone curve TC-5, as shown by the dotted line in Figure 17B, the input value 10 －３ The above 10 －２ The slope of the curve is steeper in the following range. This means that the tonal range of the dark areas (indicated by the dotted line) in the display device 40 has increased. Note that the same effect can be obtained not only with the converted image P2, but also with general SDR images when converted using the tone curve TC-5.

[0098] In each of the above embodiments, the tone curve TC has a first offset OS1 and a second offset OS2 when the brightness value of the pixels of the real-world image P1 before conversion is used as the horizontal axis and the brightness value of the pixels of the converted image P2 is used as the vertical axis, with the maximum brightness value set to 1 and the minimum brightness value set to 0. Furthermore, when this tone curve TC is expressed using a coordinate system where both axes are on a logarithmic scale, i.e., a log-log axis (common logarithm), the brightness value is 10 when the real-world image P1 is SDR. －３．５ That's 10 －１．５ This is a curve where the slope is maximum in a smaller range, and the slope asymptotically approaches zero as the luminance value approaches the maximum and minimum values. However, tone curves (TC) are not limited to this. Tone curves (TC) used for image quality conversion basically only need to have a first offset OS1.

[0099] To accurately reproduce colors across devices with different color gamuts, data compatibility in a device-independent color space is necessary. The sRGB color space is a standard color space based on a standardized specification developed to meet such requirements. The tone curves TC of the image quality conversion devices 1A to 1C described above are also basically defined based on the sRGB color space.

[0100] When this standard was established (1999), CRTs (Cathode Ray Tubes) were still common. CRTs have a significant dark current. Therefore, the black offset due to the dark current was taken into consideration, and the calculation was made to obtain correct perception without effort by adding the light emitted due to the black offset with the luminance component that is actually emitted due to the nonlinear characteristic of the CRT's γ (gamma) of 2.2 when the luminance value is input directly. When calculating the specific values, it is clear that the maximum contrast ratio assumed at that time was about 20:1, which is a very low-contrast image viewing environment by today's standards (Reference: Naoto Kawamura, Hiroaki Sugiura: "Extended Color Space and Color Management Technology III: sRGB Color Space and International Standardization," Journal of the Institute of Image Electronics Engineers of Japan, 35, 6, pp. 935-943, 2006).

[0101] However, recently, CRTs have disappeared from the market, and liquid crystal displays (LCDs) that use backlights and OLEDs (organic light-emitting diodes) that do not require backlights have become mainstream. These light-emitting displays have significantly increased maximum contrast. Furthermore, differences in the gamma characteristics of the light-emitting elements themselves can now be easily corrected using digitized brightness correction. As a result, an environment has been created in which it is easy to achieve standard characteristics, including in very low-brightness areas. Using such light-emitting displays, in dimly lit environments with little surface reflection from ambient light, it is possible to express dark images in detail with a wide dynamic range such as HDR. When expressed in 8-bit sRGB notation (see Figure 19), in an environment with a contrast ratio of approximately 300:1 or higher, it is possible to express low-brightness areas of 10 or less in 8-bit sRGB notation, which exist to a certain extent even in images taken during the day, not just in images taken at night. In order to express dark images in detail, the existence of a black offset, which is set assuming display by CRT, actually hinders the display of natural images. For this reason, in light-emitting displays, especially LCDs, there is still a fierce technological competition to prevent light leakage from the backlight (i.e., black offset in the sense of the CRT era) and to bring the black offset, as considered in the CRT era, as close to zero as possible.

[0102] Furthermore, alongside the competition to maximize the contrast of light-emitting displays as described above, there is also fierce competition to increase the brightness and saturation of the displayed image content itself (the pursuit of so-called "visual appeal"). For example, highly saturated local tone mapping (LTM), exemplified by HDR tones, is widely used. Such images also contain the low-luminance areas mentioned above.

[0103] In the image quality conversion devices 1A to 1C, as described above, a first offset OS1 is intentionally provided in the tone curve TC for the purpose of eliminating perceptual black crushing in low-luminance areas. In other words, this first offset OS1 is an offset (intercept) directly provided in the tone curve TC. It is fundamentally different from the first offset OS1 other than the image luminance value that implicitly exists and is not treated as data, such as the black offset due to the dark current of the CRT that was considered when the standard was established, which is due to the hardware characteristics of the display.

[0104] As described above, in each of the embodiments described above, the first offset OS1 is set to have a size of 10% or less of the brightness range that the converted image P2 can take. However, when the actual image P1 is an SDR or HDR original image captured with natural tones, or a CG original image based on physical norms such as ray tracing, and there is no special intention (for example, to finish in high key) as in an artistic work, regardless of the intermediate processing method, it is desirable to further optimize the size of the first offset OS1 so as not to compress the contrast ratio of the displayed image more than necessary. Below, the tone curve TC in which the first offset OS1 has been optimized will be described. In addition to the explanation of the "intermediate processing method" described above, it is also necessary to consider the intermediate processing performed inside the shooting device 30 in the process from the image sensor to the output of the actual image (first image) P1. This includes general SDR image processing such as highlight preservation by soft clipping and edge enhancement, as well as local processing represented by smartphones and general HDR tones.

[0105] Figure 20 shows tone curves TC-11 to TC-14 as examples of such tone curves TC. In this graph, the horizontal axis represents the input (sRGB normalized input) when the luminance value of each pixel in the real-world image P1 is expressed in sRGB normalized space, and the vertical axis represents the output (sRGB normalized output) when the luminance value of each pixel in the converted image P2 is expressed in sRGB normalized space. Unlike the previous tone curves TC, tone curves TC-11 to TC-14 are expressed on a linear scale rather than a logarithmic scale. TC-11 to TC-13 are tone curves TC for the conversion of SDR images. TC-14 is a tone curve TC used for the conversion of images that have undergone local processing. The tone curve TC is a curve in which, when the maximum luminance is normalized to 1 and the minimum luminance to 0, the normalized luminance value of the converted image P2 corresponding to the minimum value is greater by the first offset OS1 than the value corresponding to the minimum value of a straight line L with a slope of 1 and an intercept of 0, when both the input and output axes are on a linear scale.

[0106] In tone curve TC-11, the magnitude of the first offset OS1, i.e., the luminance value (sRGB value) in the 8-bit sRGB space, is 26. An sRGB value of 26 corresponds to 26 / 255 = 0.1020 in the graph shown in Figure 20, and the normalized magnitude of the first offset OS1 on the display is 0.0103, as shown in Figure 19. In this case, the contrast ratio is 97:1. The magnitude of the first offset OS1 in tone curves TC-12 and TC-14 is the same as the magnitude of the first offset OS1 in tone curve TC-11. On the other hand, in tone curve TC-13, the magnitude of the first offset OS1, i.e., the sRGB value, is 16. The sRGB value 16, as shown in the graph in Figure 20, is 16 / 255 = 0.0627. On an actual display, the normalized first offset OS1 is 0.0052, as shown in Figure 19. In this case, the contrast ratio is 193:1.

[0107] The tone curves TC-11 to TC-13, which convert the SDR image P1 to the second SDR image P2, are curves with a steep slope in the low-luminance range and a shallow slope in the high-luminance range. This allows for a higher contrast ratio in the low-luminance range of the converted image P2.

[0108] On the other hand, tone curve TC-14 is used when the actual image P1 is an image that has undergone local processing to increase the contrast ratio between low-luminance and high-luminance areas. Tone curve TC-14 is a curve with a steeper slope in the mid-luminance area than in the low-luminance and high-luminance areas. Tone curve TC-14 generates a converted image P2 in which the mid-luminance area is clearer.

[0109] An example of an image converted using the tone curves TC-11 to TC-14 shown in Figure 20 will be explained. Figure 21 shows the conversion of a real image (original image) P1, which is a night image taken with a relative exposure of + / - 0 EV (Exposure Value), to a converted image P2 using tone curve TC-11. Figure 21 also shows the histograms of the luminance values ​​of the real image P1 and the converted image P2. As can be seen by comparing the histograms, a first offset OS1 exists near the luminance value of 0 in the converted image P2. In the histogram of the real image P1, the distribution near the pixel value of 0 is the maximum, whereas in the histogram of the converted image P2, the pixel values ​​near the pixel value of the first image P1 are raised to above a certain value due to the first offset OS1, and the distribution of the low luminance region changes significantly, resulting in a gentle bell shape. As a result, the converted image P2 is an image in which the low luminance region is expressed in detail while maintaining naturalness.

[0110] Figure 22 shows the process of converting a real-world image P1, a nighttime image captured at a relative exposure of -2.0 EV, to a converted image P2 using the tone curve TC-12. Figure 22 also shows histograms of the luminance values ​​for both the real-world image P1 and the converted image P2. In the histogram of the real-world image P1, the distribution is maximized around pixel value 0, whereas in the histogram of the converted image P2, the pixel values ​​around pixel value 0 in the first image P1 are raised to above a certain value by the first offset OS1, and the distribution of the low-luminance region changes significantly. As a result, the converted image P2 is an image in which the low-luminance region is expressed in detail while maintaining a natural appearance.

[0111] Figure 23 shows the process of converting a real-world image (original image) P1, which is a nighttime image taken at a relative exposure of -4.0 EV, to a converted image P2 using the tone curve TC-13. Figure 23 also shows histograms of the luminance values ​​for both the real-world image P1 and the converted image P2. In the histogram of the real-world image P1, the distribution is maximized around pixel value 0, whereas in the histogram of the converted image P2, the pixel values ​​around pixel value 0 in the real-world image P1 are raised to above a certain value by the first offset OS1, and the distribution in the low-luminance region changes significantly. As a result, the converted image P2 is an image in which the low-luminance region is expressed in detail while maintaining a natural appearance.

[0112] As described above, with tone curves TC-11 to TC-13, the first offset OS1 results in a contrast ratio of approximately 100:1 to 200:1 on the display screen (see Figure 19), which increases the range of brightness that the converted image P2 can take, thus making the converted image P2 a detailed image with a wide dynamic range. In addition, with tone curves TC-11 to TC-13, the slope in the low-luminance region is maximized, which increases the contrast ratio of the low-luminance region of the second image P2.

[0113] The real-world image P1 shown in Figure 24 is an 8-bit HDR tone image generated by converting a 16-bit HDR image captured at -2.0 EV using local tone mapping so that low-luminance and high-luminance areas have high contrast. This real-world image P1 is converted to the converted image P2 using the tone curve TC-14. Figure 24 also shows histograms of the luminance values ​​for both the real-world image P1 and the converted image P2. In the histogram of the real-world image P1, the distribution is maximized around pixel value 0, whereas in the histogram of the converted image P2, the pixel values ​​around pixel value 0 in the real-world image P1 are raised to above a certain value by the first offset OS1. As a result, the converted image P2 is an image in which low-luminance areas are finely represented while maintaining naturalness. Furthermore, as shown in Figure 20, the tone curve TC-14 has a steeper slope in the mid-luminance range compared to the low-luminance and high-luminance ranges. As a result, the converted image P2 shows improved reproduction of the mid-luminance range image compared to the actual image P1.

[0114] Figure 25 shows two converted images P2: one generated using the tone curve TC-13 from an 8-bit SDR image P1 taken at night with a relative exposure of -4.0 EV, and another generated using the tone curve TC-13 from the same 16-bit HDR image P1 taken at night with a relative exposure of -4.0 EV. Figure 25 also shows the histograms of the luminance values ​​for each converted image P2. A detailed comparison of the two converted images P2 reveals that the higher the bit depth of the original image P1, the better the reproduction of dark areas. Differences in the distribution of dark areas can be seen in the histograms. Specifically, as can be seen by comparing the auxiliary lines inserted into each histogram in Figure 25, the peak shape of the darkest area is thicker in the lower converted image P2 than in the upper converted image P2, indicating that this area contains more gradations.

[0115] Figure 26 shows how a real-world image (original image) P1, which is a nighttime image captured with a relative exposure of + / - 0 EV, is converted to a converted image P2 using the tone curve TC-12. Figure 26 also shows histograms of the luminance values ​​of the real-world image P1 and the converted image P2. This converted image P2 is significantly brighter than the converted image P2 in Figure 21. In this way, by changing the tone curve TC, it is possible to generate images for night vision cameras using visible light.

[0116] The tone curves TC-11 to TC-14 described above are optimized tone curves TC according to the time of day the image was captured and the exposure conditions at the time the actual image P1 was captured, when the first image P1 is an 8-bit SDR image. Figure 27A shows the tone curves TC-13, TC-13', and TC-13'' taken at night with a relative exposure of -4.0 EV, and Figure 27B shows the tone curves TC-15, TC-15', and TC-15'' taken during the day with a relative exposure of -5.3 EV.

[0117] Figure 28A is an 8-bit SDR image taken during the daytime with a relative exposure of + / - 0 EV. In contrast, Figure 28B is a 16-bit HDR image taken during the daytime with a relative exposure of -5.3 EV. Figure 28C is the converted image P2 obtained by applying the tone curve TC-15 shown in Figure 27B to the actual image P1 in Figure 28B. Furthermore, Figure 28D is an 8-bit image P3 obtained by applying local processing to the actual image P1 in Figure 28B, and Figure 28E is an image P4 taken at the same time and with the same relative exposure using a smartphone. It is presumed that image P4 is an image that has undergone local processing within the smartphone.

[0118] As shown in Figure 28A, the real-world image P1, which is a normal SDR image, is properly exposed. Its histogram shows a wide distribution of brightness, and there is also a distribution of dark areas with brightness values ​​close to 0. Similarly, in images P3 and P4 in Figures 28D and 28E, which are thought to have used local processing, there is also a distribution of dark areas with brightness values ​​close to 0. The converted image P2 shown in Figure 28C, which was generated by converting the real-world image P1, an HDR image shown in Figure 28B with the exposure sufficiently reduced to prevent saturation, using the tone curve TC-15, is clearly of better quality than images P3 and P4, despite having a first offset O1 set in the dark areas. The size of the first offset OS1 of the tone curve TC-15 is the same as that of the tone curve TC-11.

[0119] As shown in Figure 27A, tone curves TC-13, TC-13', and TC-13'' are almost identical except for the first offset OS1. The same applies to tone curves TC-15, TC-15', and TC-15'' shown in Figure 27B.

[0120] Tone curves TC-13 and TC-15 are tone curves used for conversion to normal SDR images. Tone curves TC-13' and TC-15' are tone curves used for conversion to MDR (Moderate Dynamic Range) images. MDR images have a lower contrast ratio (gradation) than SDR images. Tone curves TC-13'' and TC-15'' are tone curves used for conversion to LDR images. LDR images have a lower contrast ratio (gradation) than MDR images. In other words, here, SDR images are defined as images with a high contrast ratio, MDR images as images with a medium contrast ratio, and LDR images as images with a low contrast ratio. MDR images are assumed to be images displayed on reflective displays or medium-quality photographic paper, while LDR images are assumed to be small images displayed, for example, as thumbnail images. Figure 29 summarizes the sRGB value, normalized value, CR, and LOG10 output values ​​of the first offset OS1 for the corresponding grayscale. As shown in Figure 29, it is desirable to increase the size of the first offset OS1 of the tone curve TC used for each conversion as the grayscale decreases, which is the threshold for obtaining the converted image P2.

[0121] Figure 30A shows the converted image P2 obtained by transforming a real-life image P1 taken at night using the tone curve TC-13, Figure 30B shows the converted image P2 obtained by transforming a real-life image P1 taken at night using the tone curve TC-13', and Figure 30C shows the converted image P2 obtained by transforming a real-life image P1 taken at night using the tone curve TC-13''. Figure 31A shows the converted image P2 obtained by transforming a real-life image P1 taken during the day using the tone curve TC-15, Figure 31B shows the converted image P2 obtained by transforming a real-life image P1 taken during the day using the tone curve TC-15', and Figure 31C shows the converted image P2 obtained by transforming a real-life image P1 taken during the day using the tone curve TC-15''.

[0122] As shown in Figures 30A to 30C and 31A to 31C, it was found that for both nighttime and daytime images, increasing the size of the first offset OS1 to a contrast ratio of 200:1 for conversion to an SDR image resulted in a more visually appealing image. Furthermore, the size of the first offset OS1 for MDR images, assuming a reflective display or medium-quality photographic paper, is approximately 32 in sRGB 8-bit representation, regardless of day or night, which translates to a contrast ratio of approximately 70:1. Therefore, it was found that even if the size of the first offset OS1 is consciously increased to at least a contrast ratio of 70:1, it is still sufficiently tolerable for general-purpose usage environments. In contrast, for LDR images intended for bright viewing environments or small-screen viewing such as thumbnail images, increasing the first offset OS1 to a contrast ratio of approximately 10:1 resulted in a clear deterioration of overall quality. Based on these results, it is desirable that the first offset OS1 be set in the tone curve TC such that the contrast ratio of the converted image P2 displayed is between 10:1 and 200:1, and that the first offset OS1 be set to decrease as the contrast ratio of the converted image P2 displayed increases. It has been found that if the size of the first offset OS1 is set to less than 1 / 200 (a value exceeding 200:1 in terms of contrast ratio), no improvement in image quality or visibility of the converted image P2 is observed.

[0123] Thus, according to the image quality conversion devices 1A to 1C of each of the above embodiments, in cases where there is no special intention such as creating an artistic work (for example, creating a high-key image), the first offset OS1 can be used to consciously raise the contrast ratio of an SDR or HDR image captured with natural tones, or a general real-life image P1 obtained through intermediate processing such as local tone mapping, above a certain value within the image content, thereby adjusting the displayed converted image P2 to an image quality with good visibility. It is desirable that the size of the first offset OS1 be set so that the contrast ratio that the displayed converted image P2 can take is between 10:1 and 200:1, and it has become clear that it is desirable that the first offset OS1 be set to decrease as the contrast ratio that the displayed converted image P2 can take increases. Thus, it is desirable that the first offset OS1 be set so that the contrast ratio of the converted image P2 does not become smaller than the required contrast ratio.

[0124] In each of the embodiments described above, the tone curve TC is used for global tone mapping. However, it is not limited to this, and the tone curve TC may also be used for local tone mapping.

[0125] Furthermore, the hardware and software configurations of the image quality adjustment devices 1A, 1B, and 1C are examples only and can be changed and modified as needed.

[0126] The core processing portion of the image quality adjustment devices 1A, 1B, and 1C, which consist of a storage unit 10 and a conversion unit 20, can be implemented using a standard computer system, rather than a dedicated system. For example, the image quality adjustment devices 1A, 1B, and 1C that perform the above processing may be configured by distributing a computer-readable recording medium (flexible disk, CD-ROM, DVD-ROM, etc.) containing a computer program for executing the above operations, and then installing the computer program on a computer. Alternatively, the computer program may be stored in a storage device on a server device on a communication network such as the Internet, and the image quality adjustment devices 1A, 1B, and 1C may be configured by downloading it from a standard computer system.

[0127] When the functions of the image quality adjustment devices 1A, 1B, and 1C are realized through a division of labor between the OS (operating system) and the application program, or through collaboration between the OS and the application program, only the application program portion may be stored on the recording medium or storage device.

[0128] It is also possible to superimpose a computer program onto a carrier wave and distribute it via a communication network. For example, a computer program could be posted on a bulletin board system (BBS) on a communication network and distributed via the network. This computer program could then be launched and executed under the control of the OS, similar to other application programs, thereby enabling the aforementioned processing.

[0129] This invention allows for various embodiments and modifications without departing from the broad spirit and scope of the invention. Furthermore, the embodiments described above are for illustrative purposes only and do not limit the scope of the invention. In other words, the scope of this invention is indicated not by the embodiments, but by the claims. Various modifications made within the scope of the claims and the equivalent scope of the meaning of the invention are considered to be within the scope of this invention.

[0130] This application claims priority based on Japanese Patent Application No. 2024-206451, filed on 27 November 2024, and incorporates the entire description, claims, and drawings of Japanese Patent Application No. 2024-206451 by reference within this specification.

[0131] This invention can be applied to adjusting the image quality of live-action images. It can also be similarly applied to photorealistic computer graphics (CG) images that faithfully reproduce brightness using techniques such as ray tracing.

[0132] 1A, 1B, 1C Image quality adjustment device, 10 Storage unit, 20 Conversion unit, 21 Image quality conversion unit, 22 Exposure time adjustment unit, 23 Environmental condition acquisition unit, 24 Tone curve adjustment unit, 30 Imaging device (imaging unit), 40 Display device (output unit), 50 Printing device (output unit), 60 CPU (Central Processing Unit), 61 Memory, 62 External storage unit, 63 Operation unit, 64 Display unit, 65 Communication interface, 66 Input / Output unit, 69 Program, 70 Internal bus, 80 Recording medium, 90 Environmental sensor, L Linear, OS1 First offset, OS2 Second offset, P1 Actual image (First image), P2 Converted image (second image), TC, TC-1, TC-2, TC-3, TC-4, TC-5, TC-11, TC-12, TC-13, TC-13', TC-13'', TC-14, TC-15, TC-15', TC-15'' Tone curve

Claims

1. An image quality adjustment device comprising: a storage unit for storing a tone curve for converting the image quality of a first image; and an image quality conversion unit for converting the image quality of the first image using the tone curve read from the storage unit, wherein the tone curve is a curve in which, when the maximum luminance is set to 1 and the minimum luminance to 0, the normalized luminance value of the converted second image corresponding to the minimum luminance is greater by a first offset than the value corresponding to the minimum luminance of a straight line with a slope of 1 and an intercept of 0, when both the input and output axes are on a linear scale.

2. The image quality adjustment device according to claim 1, wherein the first offset is set such that the contrast ratio that the displayed second image can take is 10:1 or more and 200:1 or less in the tone curve.

3. The image quality adjustment device according to claim 2, wherein in the tone curve, the first offset is set to decrease as the contrast ratio that the displayed second image can take increases.

4. The image quality adjustment device according to any one of claims 1 to 3, wherein the tone curve is a curve in which the normalized luminance value of the second image corresponding to the maximum value is smaller by a second offset than the value corresponding to the maximum value of a straight line with a slope of 1 and an intercept of 0 when both the input and output axes are on a linear scale.

5. The image quality adjustment device according to claim 4, wherein in the tone curve, the second offset has a magnitude of 10% or less of the brightness range that the second image can take.

6. The tone curve is such that when the normalized luminance value of the first image is expressed on a log-log axis (common logarithmic scale) when the gradation representation of the first image is 8-bit SDR (Standard Dynamic Range), it is 10 －３．５ That's 10 －１．５ The slope is greatest in a smaller range, and the normalized luminance value of the first image is 10 when expressed on a log-log axis (common logarithmic scale) if the grayscale representation of the first image is 16-bit HDR (High Dynamic Range). －4．５ That's 10 －１．５ The image quality adjustment device according to claim 4 or 5, wherein the slope is maximum in a smaller range, and the slope asymptotically approaches 0 as the brightness value of the first image approaches the maximum and minimum values.

7. The image quality adjustment device according to any one of claims 1 to 5, wherein the tone curve is a curve in which the slope is large in the low-luminance region and small in the high-luminance region when both the input and output axes are on a linear scale.

8. The image quality adjustment device according to any one of claims 1 to 5, wherein the first image is an image that has been transformed such that the contrast ratio between the low-luminance region and the high-luminance region is increased when both the input and output axes are set to a linear scale, the tone curve is a curve with a steeper slope in the medium-luminance region than in the low-luminance and high-luminance regions.

9. The image quality adjustment device according to any one of claims 1 to 8, wherein the memory unit stores as tone curves: a first tone curve for obtaining an image that conforms to the visual characteristics of a person viewing a landscape outdoors at night; a second tone curve for obtaining an image that conforms to the visual characteristics of a person viewing a landscape outdoors in the morning during the day; a third tone curve for obtaining an image that conforms to the visual characteristics of a person viewing a landscape outdoors in the afternoon during the day; and a fourth tone curve for obtaining an image that conforms to the visual characteristics indoors, and the magnitude of the first offset increases in the order of the second, third, fourth, and first tone curves.

10. An image quality adjustment device according to any one of claims 1 to 9, comprising: an imaging unit for capturing the first image; an output unit for outputting the second image; and an exposure time adjustment unit for adjusting the exposure time of the imaging unit according to an operation input.

11. An image quality adjustment device according to any one of claims 1 to 10, comprising: an environmental condition acquisition unit for acquiring ambient or imaging environmental conditions; and a tone curve adjustment unit for adjusting the tone curve used for converting the first image in the image quality conversion unit based on the environmental conditions acquired by the environmental condition acquisition unit, wherein the storage unit stores a plurality of tone curves corresponding to each of a plurality of different environmental conditions; and the tone curve adjustment unit either selects a tone curve corresponding to the environmental conditions acquired by the environmental condition acquisition unit from among the plurality of tone curves stored in the storage unit, or generates a tone curve corresponding to the environmental conditions acquired by the environmental condition acquisition unit by interpolating two of the plurality of tone curves.

12. The image quality adjustment device according to any one of claims 1 to 11, wherein the first image is a real-world image that has been inversely gamma corrected.

13. The image quality adjustment device according to any one of claims 1 to 12, wherein both the first image and the second image are SDR (Standard Dynamic Range) images.

14. The image quality adjustment device according to any one of claims 1 to 13, wherein the tone curve is a global tone mapping.

15. Image quality adjustment method performed by an image quality adjustment device, comprising a conversion step of converting the image quality of a first image using a tone curve for converting the image quality of a first image, wherein the tone curve is a curve in which, when the maximum luminance is set to 1 and the minimum luminance to 0, the normalized luminance value of the converted second image corresponding to the minimum luminance is greater by a first offset than the value corresponding to the minimum luminance of a straight line with a slope of 1 and an intercept of 0 when both the input and output axes are on a linear scale.

16. A program that causes a computer to function as a storage unit for storing a tone curve for converting the image quality of a first image, and an image quality conversion unit for converting the image quality of the first image using the tone curve read from the storage unit, wherein the tone curve is a curve in which, when the maximum value of luminance is set to 1 and the minimum value to 0, the normalized luminance value of the converted second image corresponding to the minimum value is greater by a first offset than the value corresponding to the minimum value of a straight line with a slope of 1 and an intercept of 0, when both the input and output axes are on a linear scale.

Images