Color adjustment method, color adjustment system, and program

Abstract

Provided is a color adjustment method capable of making hues of a background video and a video of a performer more natural in virtual production or the like.　To this end, provided is a color adjustment method executed by a computer system including a display device 10, an imaging device 20, and a computer device 30, the color adjustment method comprising a step for executing color adjustment processing on a first video 51 when the first video 51 is displayed on the display device 10, by using color information of the first video 51 and color information of a third video 54 that is a captured video obtained by capturing with the imaging device 20 a second video 52 that is a display video when the first video 51 is displayed on the display device 10.

Description

Color adjustment method, color adjustment system, and program

[0001] The present invention relates to a color adjustment method, a color adjustment system, and a program.

[0002] In recent years, as a shooting technique for content such as TV and movies, a shooting technique called virtual production has been adopted. At a shooting site using virtual production, a display device such as an LED (light-emitting diode) wall is used as the background of the shooting set, and shooting is executed with an imaging device such as a camera while an actor actually performs in front of the display device (for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2024-98589

[0004] In virtual production, the background displayed on a display device such as an LED wall and an actor can be photographed together. At this time, due to the display characteristics of the display device and the shooting characteristics of the imaging device, there is a problem that the color tone of the original background material (video) is different from the color tone of the background material in the captured image captured by the imaging device. For example, Patent Document 1 relates to a technique for adjusting the color tone of a captured image, and discloses that conversion processing is performed so that the captured image has an appropriate color tone. However, when the captured image itself is changed as in Patent Document 1 for the above problem, since the optimal value of color tone adjustment is different between the LED wall and the actor, if adjustment is made with emphasis on either the LED wall or the actor, problems such as the other not being properly color-adjusted occur.

[0005] The present invention has been made in view of such problems.

[0006] In order to solve the above problems, one aspect of the present invention is a color adjustment method executed by a computer system including a display device, an imaging device, and a computer device, the color information of a first video, and the color information of a third video which is a captured video captured by the imaging device of a second video which is a display video when the first video is displayed on the display device, the color adjustment method including the step of performing color adjustment processing on the first video when the first video is displayed on the display device using the color information of the first video and the color information of the third video.

[0007] Another aspect of the present invention is the color adjustment method described above, wherein the step of performing the color adjustment process includes deriving a correction value for the color adjustment process such that the hue of the first image matches the hue of the third image.

[0008] Another aspect of the present invention is a color adjustment method in which the step of deriving the correction values ​​is to derive correction values ​​for a plurality of colors present within the color gamut of the first image according to a specific algorithm, and then use the derived correction values ​​to derive correction values ​​for colors other than the plurality of colors by interpolation.

[0009] Another aspect of the present invention is the color adjustment method described above, wherein the particular algorithm includes searching for color information of the second image in which the hue of the third image matches the hue of the first image.

[0010] Another aspect of the present invention is a color adjustment method described above, wherein the step of deriving the correction values ​​is to derive a first correction value for six primary colors including the three primary colors of light (R, G, B) and the three primary colors of pigment (Y, C, M); to derive a second correction value using the first correction value for colors that are in the same position as the six primary colors in the triangle representing the color gamut on the xy chromaticity diagram; to derive a third correction value using at least the second correction value for colors that are on the sides of the triangle representing the color gamut on the xy chromaticity diagram; and to derive a fourth correction value using at least the third correction value for colors that are located inside the triangle representing the color gamut on the xy chromaticity diagram.

[0011] Another aspect of the present invention is the color adjustment method described above, wherein the step of performing the color adjustment process includes correcting the color information of the second video using the correction value derived in the step of deriving the correction value.

[0012] Another aspect of the present invention is the color adjustment method described above, wherein the correction values ​​derived in the step of deriving the correction values ​​are used and stored in the computer device as 3DLUT data.

[0013] Another aspect of the present invention is a computer system that performs the color adjustment method described above.

[0014] Another aspect of the present invention is a program for causing a computer system to perform the above-described color adjustment method.

[0015] Another aspect of the present invention is a computer-readable recording medium that stores a program for causing a computer system to execute the above-described color adjustment method.

[0016] This is a diagram illustrating an example of the configuration of a color adjustment system according to one embodiment of the present invention. This is a diagram illustrating an overview of a color adjustment system according to one embodiment of the present invention. This is a diagram illustrating an example of the hardware configuration of the computer device 30. This is an example of a flowchart showing the processes performed in the color adjustment system according to one embodiment of the present invention. This is an example of a flowchart showing the processes performed in the color adjustment system according to one embodiment of the present invention. This is an example of a flowchart showing the processes performed in the color adjustment system according to one embodiment of the present invention. This is a diagram illustrating an example of step S110. This is a diagram illustrating an example of step S112. This is an image diagram illustrating the flow of deriving correction values ​​in a color adjustment method performed by the color adjustment system according to one embodiment of the present invention. This is an image diagram of steps 2 to 4 of Figure 7. This is a diagram illustrating an example of step 2. This is a diagram illustrating the search method in step 3. This is a diagram illustrating the first pattern of step 3. This is a diagram illustrating the second pattern of step 3. This is a diagram illustrating the third pattern of step 3. This is a diagram illustrating the fourth pattern of step 3. This is a diagram illustrating an example of deriving correction values ​​in step 4. This is a diagram illustrating another example of deriving correction values ​​in step 4. This figure shows another example of deriving correction values ​​in step 4. This is an image diagram showing the colors for which correction values ​​were derived in steps 1 to 4. This is an image diagram of 3DLUT generation. This figure shows an example of a functional block diagram of the computer device 30. This figure illustrates the case where multiple monochrome images are combined into a single display image.

[0017] Embodiments of the present invention will be described in detail below with reference to the drawings.

[0018] (Configuration of the color adjustment system) Figure 1 is a diagram showing an example of the configuration of the color adjustment system according to this embodiment. The color adjustment system 1 according to this embodiment may be configured to include a display device 10, an imaging device 20, and a computer device 30. The color adjustment system 1 according to this embodiment will be described below assuming that it is a virtual production system applicable to the shooting of content such as television and movies (however, this is just an example and not limited thereto). The color adjustment system 1, for example, displays the background of the shooting scene on the display device 10, and the imaging device 20 shoots the performer 5 acting in front of the display device 10.

[0019] The display device 10 is, for example, an LED display (but is not limited thereto). The imaging device 20 is, for example, a camera for capturing content such as television or movies. The computer device 30 may be connected to the display device 10 and the imaging device 20 so as to enable the transmission and reception of data by various wired or wireless communications. The computer device 30 may perform processing according to the color adjustment method of this embodiment using various data received from the display device 10 or the imaging device 20. The computer device 30 may also transmit to the display device 10 or the imaging device 20 the results of the processing according to the color adjustment method of this embodiment, as well as control data for performing necessary controls related to display or shooting. Furthermore, it is desirable, but not limited to, that data be transmitted and received between the computer device 30 and the display device 10 and the imaging device 20 via a wired or wireless network. For example, data may be transmitted via a computer-readable recording medium.

[0020] The shooting environment in the color adjustment system 1 is preferably a darkroom (but not limited to this). Also, the white balance and black balance of the imaging device 20 should be adjusted at the start of shooting. Other settings should be adjusted to suit the photographer's intentions. Furthermore, in this embodiment, it is preferable that automatic functions such as auto white balance and auto exposure are turned off in the imaging device 20.

[0021] (Overview of the Color Adjustment System) Figure 2 is a diagram illustrating the overview of the color adjustment system 1 according to this embodiment. The background material (video) 51 is displayed in color gamut X on the display device 10. The imaging device 20 captures the displayed video 52 of the background material 51 displayed on the display device 10 (and the performer 5 if one exists; the performer 5 is omitted in Figure 2). Captured video 53 is generated by the capture by the imaging device 20. Depending on the settings of the imaging device 20, the captured video 53 may be in a color gamut Y different from color gamut X. In that case, a color gamut conversion process is performed on the captured video 53 to convert it into a video 54 in color gamut X.

[0022] Generally, due to the characteristics of the display device 10 and / or the imaging device 20, the colors of image 51 and image 54 will be different. The color adjustment system 1 of this embodiment aims to match the colors of image 51 and image 54. More specifically, the color adjustment system 1 of this embodiment aims to match the hue of image 54 to the hue of image 51. To this end, in order to determine what color the image 52 displayed on the display device 10 should be, the color adjustment system 1 searches for an appropriate color for image 52 while changing the color of image 52.

[0023] Then, once the search is complete, the computer device 30 performs a color adjustment process on the image 52 so that when the image 51 is displayed on the display device 10, it will have the color determined by the search. In this embodiment, as an example, the computer device 30 applies a 3DLUT (3-Dimensional Lookup Table) for color adjustment when displaying the image 51 on the display device 10.

[0024] Furthermore, when performing the search, it is sufficient that the color gamuts of video 51 and video 54 match; therefore, if the color gamut of the captured video 53 is color gamut X, no color gamut conversion processing is required for video 53. The concept of "hue matching" will be explained below.

[0025] First, the hue θ of each image is determined as follows. (In the following, RGB values ​​are normalized.) Premise: 0≦R≦1, 0≦G≦1, 0≦B≦1 (For ITU-R BT.709 (standard)) Cb = -0.1146R - 0.3854G + 0.5000B Cr = 0.5000R - 0.4542G - 0.0458B (For ITU-R BT.2020 (standard)) Cb = -0.1396R - 0.3604G + 0.5000B Cr = 0.5000R - 0.4598G - 0.0402B (For other color gamut standards, conversion is performed according to each standard.)

[0026] ...Formula (1)

[0027] In this embodiment, "hue matching" or "hue matching" means that the angles from the center of the vector display used in the waveform monitor match. While this application uses an example of a vector display in a waveform monitor for explanation, it is not limited to this. The color adjustment system and color adjustment method according to this embodiment can be realized using elements different from the vector display of a waveform monitor. For example, "hue" in this embodiment may refer not only to a vector display but also to Hue. Furthermore, those skilled in the art will understand that "hue matching" or "hue matching" includes not only a perfect match but also acceptable deviations or errors within a range that does not hinder identification, as well as approximations through interpolation as shown in this embodiment.

[0028] Furthermore, the images 51, 52, 53, and 54 may be still images or moving images. Also, although this embodiment is described assuming the use of 3DLUT, other data formats may be used.

[0029] (Hardware Configuration) The configuration of the computer device 30 described above can be realized with a hardware configuration similar to that of a general computer device. Figure 3 is a diagram showing an example of the hardware configuration of the computer device 30. The computer device 30 shown in Figure 3 includes, as an example, a processor 31, RAM (Random Access Memory) 32, ROM (Read Only Memory) 33, a built-in hard disk drive 34, removable memory 35 such as an external hard disk drive, CD, DVD, USB memory, memory stick, SD card, etc., input / output user interface 36 (keyboard, mouse, touch panel, speaker, microphone, lamp, etc.) for the user to exchange data with the computer device 30, a wired / wireless communication interface 37 that can communicate with the display device 10, imaging device 20, or other devices, and a display 38 (which may be shared with the display device 10). The functions of the computer device 30 according to this embodiment can be realized, for example, by the processor 31 reading programs pre-stored in the hard disk drive 34, ROM 33, removable memory 35, etc., into memory such as RAM 32, and executing the programs while appropriately reading the aforementioned data necessary for processing from the hard disk drive 34, ROM 33, removable memory 35, etc. Note that the hardware configuration shown in Figure 3 is merely an example and is not limited thereto. For example, some components of the computer device 30 may be configured using a PLD (programmable logic device) such as an FPGA (field-programmable gate array). Furthermore, some components of the computer device 30 can be implemented entirely in hardware, or through a combination of a computer and software.

[0030] (Flowchart) Figures 4A to 4C are examples of flowcharts showing the processes performed in the color adjustment system 1 according to this embodiment. The processes performed in the color adjustment system 1 will be described below with specific examples. For the sake of simplicity in this explanation, RGB values ​​will be described using an 8-bit representation.

[0031] The computer device 30 determines the background material (video) 51 to be displayed on the display device 10 (step S102). For example, the computer device 30 may determine the background material 51 by receiving input from the user specifying the background material 51. The computer device 30 also determines the color gamut of the determined background material 51. Various color gamuts are conceivable, such as ITU-R BT. 709 and ITU-R BT. 2020. Possible methods for determining the color gamut of the background material 51 include the operator of the computer device 30 inputting the color gamut verbally communicated by the creator of the background material 51 to the computer device 30, or reading metadata containing information about the color gamut of the background material 51 (known software can be used).

[0032] Next, the computer device 30 prepares a monochrome image (video 51) of R (red), G (green), or B (blue), or Y (yellow), C (cyan), or M (magenta) with the same color gamut as the background material 51 determined in step S102 (step S104). The computer device 30 can prepare the monochrome image by, for example, generating the monochrome image itself, or by receiving the monochrome image from another device via a network or storage medium. When the monochrome images of R, G, B, Y, C, and M are represented by RGB values, for example, they can be represented as R (255, 0, 0), G (0, 255, 0), B (0, 0, 255), Y (255, 255, 0), C (0, 255, 255), and M (255, 0, 255) using 8 bits. In this embodiment, these six RGB values ​​will be described as six primary colors. Furthermore, while this explanation assumes a maximum value of 255 and a minimum value of 0, this is not an exhaustive rule. For example, the maximum and minimum values ​​may differ depending on the video standard. For instance, a case where the maximum value is 240 and the minimum value is 10 is conceivable.

[0033] Next, the display device 10 displays an undisplayed monochrome image from among the monochrome images prepared in step S104 (step S106). When step S106 is performed for the first time, the monochrome image to be displayed may be determined by the computer device 30 randomly selecting one of the monochrome images, or by the user specifying it through input operations to the computer device 30, etc.

[0034] Next, the computer device 30 acquires the captured image (video 53) taken by the imaging device 20 from the monochrome image (video 52) displayed on the display device 10 via a network or storage medium, and determines the color gamut of video 53. If the color gamut of video 53 is not the same as that of video 51, the computer device 30 performs a color gamut conversion process on the captured image (video 53) to generate video 54 with the same color gamut as video 51 (step S108). If the color gamut of video 51 and the color gamut of video 53 are the same, the color gamut conversion process is unnecessary (however, for the sake of convenience in the description, video 53 on which the color gamut conversion process is not performed is also described as "video 54" in the sense that it is a captured image with the same color gamut as video 51).

[0035] Next, all or part of the RGB values ​​of the video 54 are obtained, and a method for searching for the RGB values ​​of the video 52 is determined (step S110). More specifically, for example, if the monochrome image 51 is R, G, and B, the computer device 30 determines the corresponding values ​​in the video 54 for the R, G, or B values ​​that are "0" in the video 51. For example, if the monochrome image 51 is G (green), the value of G is "255" and the values ​​of R and B are "0", so the computer device 30 obtains the R and B values ​​of the video 54 (known techniques can be used). The computer device 30 then decides that the search method involves increasing the smaller of the R and B values ​​of the video 54, and performing a search to determine what kind of correction should be made when outputting the video 52 from the video 51 so that the hues of the video 51 and the video 54 match. Furthermore, if the monochrome image 51 is Y, C, M, the computer device 30 determines the corresponding values ​​in the image 54 for R, G, or B, which have a value of "255" in the image 51. For example, if the monochrome image 51 is Y (yellow), the values ​​of R and G are "255" and the value of B is "0", so the computer device 30 obtains the R and G values ​​of the image 54. The computer device 30 then decides to search for correction values ​​for the image 52 while decreasing the larger of the R and G values ​​in the image 54.

[0036] Figure 5 illustrates an example of step S110. The processing in step S110 differs depending on whether the video 51 is a monochrome image of R, G, B or a monochrome image of Y, C, M. Figure 5(1) illustrates an example where the video 51 is a monochrome image of R, G, or B. The table in (1) shows, as an example, that the RGB value of video 51 is (0, 255, 0) (i.e., video 51 is G (green)), the RGB value of video 52 at this time is (0, 255, 0), and the RGB value of video 54 is (60, 220, 10) (from here on, tables of the same format as (1) will have the same meaning). In this embodiment, the RGB value of video 52 is not the apparent RGB value of video 52 displayed on the display device 10, but the RGB value as output data when video 52 is displayed. That is, in the initial state of the search, the RGB value of video 51 = the RGB value of video 52.

[0037] In case (1), the values ​​of R and B in video 51 are the same (i.e., the value is "0"), and the corresponding R and B values ​​in video 54 are "60" and "10", respectively. In other words, R > B in video 54. The color adjustment system of this embodiment aims to match the hues of video 51 and video 54, and based on the above equation (1), in (1), the hues of video 51 and video 54 match when video 54 becomes R = B. Therefore, in the subsequent processing, the computer device 30 repeatedly changes the value of B in video 52 so that the value of B in video 54 gradually increases and approaches the value of R. In this example, the reason why the value of B in video 54 is controlled to gradually increase because R > B in video 54 is that the R value in video 52 is the minimum value of "0", and it is not possible to decrease the R value of video 52 so that the R value of video 54 decreases. Therefore, in case (1), the computer device 30 searches for RGB values ​​for video 52 such that the R value ≈ B value of video 54 by repeatedly increasing the R or B value of video 52 so that the smaller of the R and B values ​​of video 54 gradually increases.

[0038] Furthermore, Figure 5(2) illustrates an example where the image 51 is a monochromatic image of Y, C, or M. The table in (2) shows that, as an example, the RGB value of image 51 is (255, 255, 0) (i.e., image 51 is Y (yellow)), the RGB value of image 52 is (255, 255, 0), and the RGB value of image 54 is (200, 220, 30). In case (2), the values ​​that are the same in image 51 (i.e., the value is "255") are R and G, and the corresponding R and G values ​​in image 54 are "200" and "220", respectively. In other words, in image 54, G > R. The goal is to match the hues of image 51 and image 54, as in (1). Based on the above equation (1), the hues of image 51 and image 54 match in (2) when image 54 has R = G. Therefore, in the subsequent processing, the computer device 30 repeatedly changes the G value of video 52 so that the G value of video 54 gradually decreases and approaches the R value. In this example, the reason why the G value of video 54 is controlled to gradually decrease because G > R in video 54 is that the R value of video 52 (video 51) is the maximum value of "255", and it is not possible to increase the R value of video 52 so that the R value of video 54 increases. Therefore, in case (2), the computer device 30 searches for the RGB values ​​of video 52 such that the R value of video 54 is approximately equal to the G value by repeatedly gradually decreasing the G or R value of video 52 so that the larger of the G and R values ​​of video 54 gradually decreases.

[0039] Next, the computer device 30 causes the display device 10 to display the image 52 while changing its RGB values ​​according to the search method determined in step S110 (step S112). Figure 6 is a diagram illustrating an example of step S112. In this example, we will continue to explain the case of (1) in Figure 5 as an example. In step S112, a search method was determined in which the value of B in the image 52 is gradually increased. Therefore, the computer device 30 controls the display device 10 so that the image 52 is displayed on the display device 10 while gradually increasing the value of B in the image 52. Figure 6 shows that the computer device 30 repeatedly displays the image 52 on the display device 10, starting with an initial value of (0, 255, 0) and increasing the value of B by "10". More specifically, the computer device 30 controls the display device 10 to sequentially generate (or obtain from other computer devices, etc.) images 52 with RGB values ​​of (0, 255, 10), (0, 255, 20), (0, 255, 30), etc., and to output them for display.

[0040] First, the display device 10 displays an image 52 with RGB values ​​of (0, 255, 10). The imaging device 20 captures the image 52 displayed on the display device 10 and transmits the image data of the captured image 53 to the computer device 30 via the network (or the image data may be passed to the computer device 30 via a storage medium; the same applies hereinafter). The computer device 30 converts the color gamut of the captured image 53 as needed to generate an image 54. Alternatively, the color gamut conversion of the image 53 is performed by another color gamut conversion device, a display medium for the image 54, or other device or software to generate the image 54. The computer device 30 determines the RGB values ​​of the image 54. In this example, the RGB values ​​of the image 54 at this time are (60, 255, 15). Next, the computer device 30 generates an image 52 with RGB values ​​of (0, 255, 20) and displays it on the display device 10. The imaging device 20 captures the displayed image 52 and transmits the image data of the captured image 53 to the computer device 30. The computer device 30 determines the RGB values ​​of the color-gamut-converted image 54 as needed. In this example, the RGB values ​​of the image 54 at this time are (60, 255, 20). Similarly, the computer device 30 displays the RGB values ​​of the image 52 on the display device 10 while changing them to (0, 255, 30), (0, 255, 40), (0, 255, 50), etc., and searches for the timing when the hue shift of the image 54 reverses or when the hue shift of the image 54 disappears. Here, the timing when the hue shift reverses is the timing when R B, based on the above equation (1). Also, the timing when the hue shift of the image 54 disappears is the timing when R = B, based on the above equation (1).

[0041] Here, when the RGB values of video 52 change from (0, 255, 70) to (0, 255, 80), the RGB values of video 54 change from (60, 255, 56) to (60, 255, 62), and R < B. From this, it can be seen that there exists an RGB value of video 52 between the RGB values (0, 255, 70) and (0, 255, 80) of video 52 such that the RGB value of video 54 becomes (60, 255, 60). Thus, when the computer device 30 determines that the hue shift of video 54 has reversed (i.e., changed from R > B to R < B) or the hue shift of video 54 has disappeared (i.e., R = B) (step S114), it derives a correction value for color adjustment for video 52 (step S116).

[0042] The correction value for color adjustment for video 52 can be calculated, for example, as follows: Assuming that the hue of each video is obtained as in the above formula (1), θ1 = (hue of video 51 - hue of video 54 (A)) When θ1 > 180, θ1 = 360 - θ1 When θ1 < -180, θ1 = 360 + θ1 θ2 = (hue of video 54 (B)) - (hue of video 54 (A)) When θ2 > 180, θ2 = 360 - θ2 When θ2 < -180, θ2 = 360 + θ2 Here, n: the value by which the RGB value of video 52 changes when the hues of video 51 and video 54 match A: immediately before the hue shift reverses B: when the hue shift has reversed Then,

[0043] In the above formula, the "value by which the RGB value of video 52 changes" refers to the value that changes during the search among the R, G, and B values of video 52. In the example of FIG. 6, it is the value of B.

[0044] For the example of FIG. 6, calculating n according to the above formula is as follows.

[0045]

[0046] That is, when the RGB value of video 52 is (0, 255, 76.6), the hues of video 51 and video 54 will match. When the RGB value of video 54 becomes R = B by the search described in FIG. 6, the value of B of video 52 at that time may be used as the correction value.

[0047] Incidentally, when handling RGB values in 8 bits (0 to 255), since the values need to be integers, for example, the nearest integer may be used as the correction value. Although it is described in 8 bits for simplification of the explanation this time, in the present embodiment, since the RGB values are normalized to 0 to 1 before processing, the correction value calculated by the above formula is used as it is.

[0048] In the example of FIG. 6, the value of B of the video 52 is changed by "10", but it is not limited to this. The width of the change may be made smaller or larger.

[0049] The computer device 30 repeats steps S106 to S116 in the same manner for each of the R, G, B, Y, C, and M monochromatic images to derive correction values for all the monochromatic images (step S118).

[0050] Here, FIG. 7 is an image diagram for explaining the flow of derivation of correction values in the color adjustment method executed by the color adjustment system 1 according to the present embodiment. The color adjustment method according to the present embodiment derives correction values for each color in the order of six primary colors (step 1), a color at the same position (same coordinates) as the six primary colors in a triangle indicating a color gamut on the xy chromaticity diagram (step 2), a color on the side of the triangle indicating the color gamut on the xy chromaticity diagram (step 3), and a color located inside the triangle indicating the color gamut on the xy chromaticity diagram (step 4). As shown in FIG. 7, the color adjustment method according to the present embodiment is finally an image in which corrections are derived for each vertex of the triangle having G, B, and R as vertices, colors located on each side, and colors located inside the triangle. The processing up to step S118 described above has completed step 1 of FIG. 7. Hereinafter, steps 2 to 4 are executed.

[0051] In steps 2 to 4, in order to derive each correction value, the computer device 30 determines the interval of the color (hereinafter referred to as "measurement color") that is the target of the deriving of the correction value in the video 51. Here, as an example, it is determined to be "85". The computer device 30 also determines the interval of change of the value to be changed during the search for each measurement color in the video 52. Here, as an example, it is determined to be "5" (step S120). Note that each data determined in step S120 may be determined, for example, by the computer device 30 receiving setting input from the user.

[0052] Here, Figure 8 is an illustrative diagram of steps 2 to 4 of Figure 7. As explained in Figure 7, the general order of deriving the correction values ​​is as follows: after step 1, (1) the colors that are in the same position (same coordinates) as the six primary colors in the triangle that shows the color gamut on the xy chromaticity diagram (step 2), (2) the colors that are on the sides of the triangle that shows the color gamut on the xy chromaticity diagram (step 3), and (3) the colors that are located inside the triangle (step 4). Although the triangle that shows the color gamut on the xy chromaticity diagram illustrated in Figure 7 is two-dimensional, in reality, there are colors in the brightness direction that are not represented on the xy chromaticity diagram, and it will be understood by those skilled in the art that correction values ​​for those colors are derived in steps 2 to 4.

[0053] Figure 9 illustrates an example of (1) step 2. In case (1), first, correction values ​​are searched in the same way as in step 1 for some of the colors that are at the same position (same coordinates) as the six primary colors in the triangle that shows the color gamut on the xy chromaticity diagram between (0,0,0) and each primary color. Then, correction values ​​are interpolated for the other colors using the searched correction values. Figure 9 illustrates, as an example, the case in which correction values ​​are derived for a color located between (0,0,0) and G(0,255,0). As described above, in step S120, the computer device 30 determined the interval of the measurement colors in the image 51 to be "85" and the interval of change of the value to be changed during the search for each measurement color in the image 52 to be "5". Therefore, in this example, the measurement colors of the image 51 are (0,85,0) and (0,170,0). Here, when deriving a correction value for the measured color (0,85,0) of the image 51, the correction value is derived using the same search method as the one used to search for the correction value of (0,255,0) in step 1. In the example of Figure 6 described above, the correction value for (0,255,0) was searched while increasing the value of B, so similarly, the search for (0,85,0) is performed while gradually increasing the value of B (step S124). Also, since the interval for changing the value to be changed during the search was determined to be "5" in step S120, the computer device 30 displays the image 52 on the display device 10 while increasing the value of B by "5" each time. From here on, it is the same as in step 1. That is, while displaying the image 52 on the display device 10 while increasing the value of B by "5" each time, the imaging device 20 sequentially captures the displayed image 52 (step S126). The computer device 30 sequentially checks the R and B values ​​of the captured image 54 and determines the point in time when R = B or R < B based on the above formula (1) (step S128). If the computer device 30 determines that R = B or R < B has occurred, it derives a correction value for color adjustment of the video 52 (step S130). Similarly, the computer device 30 derives a correction value for the video 51, which has RGB values ​​of (0, 170, 0), by performing steps S122 to S130.The computer device 30 derives correction values ​​for the images 51 with RGB values ​​(0,0,0), (0,85,0), and (0,170,0) (and (0,255,0)) (step S132), and then derives correction values ​​by linear interpolation for the colors from RGB values ​​(0,0,0) to (0,255,0) for which no correction values ​​have been derived (step S134). This completes the process in step 2.

[0054] Similarly, steps S122 to S134 are executed for steps 3 and 4.

[0055] Here, we will explain how to derive the correction values ​​in step 3. In step 3, correction values ​​are derived for colors that lie between the colors for which correction values ​​were obtained in step 2. For example, correction values ​​are derived for colors located between G(0,255,0) and Y(255,255,0), for colors located between (0,85,0) and (85,85,0), and for colors located between (0,170,0) and (170,170,0). However, the search interval in step 2 and the search interval in step 3 may be different. For example, if the interval was "85" in step 2, it may be "51" in step 3. In that case, in step 3, correction values ​​are derived for colors located between G(0,255,0) and Y(255,255,0), for colors located between (0,51,0) and (51,51,0), for colors located between (0,102,0) and (102,102,0), and so on. The following explanation of step 3 will focus on the case where correction values ​​are derived for colors located between G(0,255,0) and Y(255,255,0).

[0056] In Figure 10, as an example, the correction value for image 52 for G(0,255,0) in image 51 is (0,255,77), indicating that the search for this correction value was conducted while increasing the B value. Similarly, the correction value for image 52 for Y(255,255,0) in image 51 is (255,220,0), indicating that the search for this correction value was conducted while decreasing the G value. In such cases, it can be predicted that for colors between G(0,255,0) and Y(255,255,0), the correction value will be searched by decreasing the G value and increasing the B value. In other words, it is possible to predict the trend in the method of searching for correction values ​​for colors on the sides of the triangle from how the correction values ​​for each primary color were searched.

[0057] During the search, the RGB values ​​of the image 52 are either "changing", "maintaining '255'", or "maintaining '0'". For example, when searching for a correction value for G(0,255,0) in step 1, the search is performed while increasing the smaller of the R value or B value (see Figure 5). Also, when searching for a correction value for Y(255,255,0) in step 1, the search is performed while decreasing the larger of the R value or G value (see Figure 5). Therefore, when attempting to perform a search between G and Y, the search tendencies for colors on the G-Y edge should be predicted into the following four patterns: Pattern 1: The search was performed by increasing the R value of G and decreasing the R value of Y. Pattern 2: The search was performed by increasing the R value of G and decreasing the G value while maintaining the R value of Y at "255". Pattern 3: The search was performed by increasing the B value while maintaining the R value of G at "0" and decreasing the R value of Y. Pattern 4: The search was performed by increasing the B value while maintaining the R value of G at "0" and decreasing the G value while maintaining the R value of Y at "255". The same four patterns can be assumed when deriving correction values ​​for colors on the edge between other primary colors of light and primary colors of pigment. When the computer device 30 performs the derivation of correction values ​​in step 3, it first determines which of the above patterns the derivation of correction values ​​to be performed corresponds to, and then performs the derivation of correction values ​​in each pattern, as described later.

[0058] The following explains, as an example, how to derive correction values ​​for the color on the edge between G and Y.

[0059] (First Pattern) Figure 11 illustrates the first pattern of step 3. In the first pattern, the computer device 30 sets the correction value for colors at the same position (same coordinates) as the three primary colors of light (R, G, B) in the triangle representing the color gamut on the xy chromaticity diagram to the maximum value, and the correction value for colors at the same position (same coordinates) as the three primary colors of pigment (Y, C, M) in the triangle representing the color gamut on the xy chromaticity diagram to the minimum value, and performs a search for the color of the image 52 such that the hues of image 51 and image 54 match while changing the RGB values ​​of the image 52 within the range between this minimum and maximum value. In this example, the correction value of the image 52 for image 51 with G (0, 255, 0) is (50, 255, 0), and the correction value of the image 52 for image 51 with Y (255, 255, 0) is (220, 255, 0). Note that the RGB values ​​of image 54 are determined so that the hue matches that of image 51, and details are omitted in this example (the same applies to the explanation of patterns 2 to 4 below). In this case, the computer device 30 derives a correction value by displaying on the display device 10 the R value of image 52 at intervals of "5", which is the interval of change determined in step S120, for each interval of the measurement color determined in step S120 (in this example, the interval of the measurement color is set to "128" for simplicity of explanation; the same applies to patterns 2 to 4). The R value of image 52 is changed at intervals of "5", which is the interval of change determined in step S120, to (55, 255, 0), (60, 255, 0), (65, 255, 0), ... (210, 255, 0), (215, 255, 0). Through this process, correction values ​​are derived for several colors located between G and Y. Note that the measurement interval in this example is "128", but the measurement interval (search interval) here may be a different value from the search interval in each of the search scenes described above. The interval between each of these searches may be determined by the computer device 30 receiving user input. (The same applies hereafter.)

[0060] (Second Pattern) Figure 12 illustrates the second pattern of step 3. In the second pattern, first, the computer device 30 searches for correction values ​​for colors that are at the same position (same coordinates) as the three primary colors of light (R, G, B) in the triangle representing the color gamut on the xy chromaticity diagram, and calculates the absolute difference between the primary color and the correction value for any of the R, G, or B values ​​whose values ​​were changed in the image 52. Similarly, the computer device 30 searches for correction values ​​for colors that are at the same position (same coordinates) as the three primary colors of pigment (Y, C, M) in the triangle representing the color gamut on the xy chromaticity diagram, and calculates the absolute difference between the primary color and the correction value for any of the R, G, or B values ​​whose values ​​were changed in the image 52. The computer device 30 then compares the two calculated absolute values ​​and first derives a correction value by linear interpolation for the R, G, or B value with the smaller absolute value. Then, for the R, G, or B value with a larger absolute value of the correction, it searches for a color for the image 52 that matches the hue of the image 51 and the image 54 while changing the R, G, or B value of the image 52. If the two calculated absolute values ​​are the same, the computer device 30 first interpolates the R, G, or B values ​​of the three primary colors of pigment by linear interpolation, and then interpolates the correction value by linear interpolation for the R, G, or B values ​​of the three primary colors of light.

[0061] In the example in Figure 12, the correction value for video 52 for the G value of video 51 is (0, 255, 0), and it is assumed that this correction value was searched while increasing the R value. In this case, the absolute difference between the two R values ​​is 50. Also, the correction value for video 52 for the Y value of video 51 is (255, 235, 0), and it is assumed that this correction value was searched while decreasing the G value. The absolute difference between the two G values ​​is 20. Since 50 > 20, the G value of the measured color (128, 255, 0) is interpolated between "235" and "255" by linear interpolation, and the G value is determined to be "245". Subsequently, for the R value of (128, 255, 0), a search is performed for the R value of video 52 that matches the hue of video 51 and video 54 while changing the R value of video 52.

[0062] (Third Pattern) Figure 13 illustrates the third pattern of step 3. In the case of the third pattern, the search is performed using the same method as in the second pattern. In the example in Figure 13, the correction value for video 52 for G (0, 255, 0) of video 51 is (0, 255, 40). It is also assumed that this correction value was searched while increasing the B value. The difference (absolute value) between the two B values ​​is 40. Also, the correction value for video 52 for Y (255, 255, 0) of video 51 is (220, 255, 0). It is also assumed that this correction value was searched while decreasing the R value. The difference (absolute value) between the two R values ​​is 35. Since 40 > 35, the R value of the measured color (128, 255, 0) is interpolated between "0" and "220" by linear interpolation, and the R value is determined to be "110". Subsequently, for the B value (128, 255, 0), a search is performed to find the B value of video 52 that matches the hue of video 51 and video 54 while increasing the B value of video 52.

[0063] (Fourth Pattern) Figure 14 illustrates the fourth pattern of step 3. In the case of the fourth pattern, the search is performed using the same method as in the second pattern. In the example in Figure 14, the correction value for video 52 for G (0, 255, 0) of video 51 is (0, 255, 40). It is also assumed that this correction value was searched while increasing the B value. The difference (absolute value) between the two B values ​​is 40. Also, the correction value for video 52 for Y (255, 255, 0) of video 51 is (255, 195, 0). It is also assumed that this correction value was searched while decreasing the G value. The difference (absolute value) between the two G values ​​is 60. Since 40 < 60, the B value of the measured color (128, 255, 0) is interpolated between "0" and "40" by linear interpolation, and the B value is determined to be "20". Subsequently, a search is performed for the G value of video 52 such that the hues of video 51 and video 54 match, while decreasing the G value of video 52 for the G value of (128, 255, 0).

[0064] The computer device 30 determines whether the measured color, which is the target of the derivation of the correction value in step 3 (step S136), falls into one of the first to fourth patterns, and performs the same processing as in steps S122 to S134 as appropriate. For example, the computer device 30 displays the measured color on the display device 10 (step S122) and performs the search for the correction value while changing the color of the image 52 according to the algorithm described above (steps S126 to S130). In addition, for colors for which a correction value has not been derived, the correction value is derived by linear interpolation (step S134). As a result, the derivation of the correction value for each primary color (step 3) in Figure 7 is completed.

[0065] Next, we will explain how the correction values ​​are derived in step 4. In step 4, the correction values ​​are searched for colors located inside the triangle that represents the color gamut on the xy chromaticity diagram.

[0066] Figure 15 shows an example of how the correction value is derived in step 4. In this example, the correction value (0,255,77) for G(0,255,0) is found by increasing the B value. In this embodiment, as described above, the white balance and black balance of the imaging device 20 are adjusted in advance, so it is not necessary to find the correction value for colors where all RGB values ​​are the same (the computer device 30 has the information that the correction value is "0"). In Figure 16, when the RGB values ​​are all the same, (200,200,200) in the figure, the correction value is 0, so the RGB value of the correction value for image 52 is the same as the RGB value of image 51 (the same applies in subsequent examples). In this example, since the correction value (0,255,77) was found by increasing the B value, the computer device 30 decides to search for the correction value between (0,255,0) and (255,255,255) by increasing the B value.

[0067] Figure 16 shows another example of how the correction value is derived in step 4. In this example, the correction value (100, 160, 30) for (100, 200, 0) is found using the fourth pattern (see Figure 14). In this example, the computer device 30 searches for the correction value between (100, 160, 30) and (200, 200, 200) in the same way as the search method for the correction value (100, 160, 30), by linearly interpolating the B value and then searching while changing the G value of the image 52.

[0068] Figure 17 shows another example of how the correction value is derived in step 4. In this example, the correction value (20, 90, 150) for (0, 100, 150) is searched by linearly interpolating the G value and then increasing the R value of the image 52. In this example, the computer device 30 also searches for the correction value between (0, 100, 150) and (150, 150, 150) in the same way as the search method for the correction value (0, 100, 150), by linearly interpolating the G value and then increasing the R value of the image 52.

[0069] This process is similarly performed for multiple colors located between the colors on the sides of the triangle and the white point, thereby completing the derivation of the correction value in step 4. Here, "white point" refers to the point closest to "white" (255, 255, 255). For example, in the example in Figure 16, the white point is (200, 200, 200), and in the example in Figure 17, the white point is (150, 150, 150).

[0070] Figure 18 is an illustrative diagram showing the colors for which correction values ​​have been derived up to this point. Once steps 1 to 4 are performed and the derivation of correction values ​​for the six primary colors is complete, as shown in Figure 15, the derivation of correction values ​​will be complete for multiple colors on the sides of the color gamut triangle and on the line segments connecting multiple colors located on the sides to the white point. Note that the number of line segments connecting multiple colors located on the sides to the white point in Figure 18 depends on the interval of the measurement values ​​(Figure 18 is merely an illustrative diagram).

[0071] The computer device 30 derives correction values ​​for colors other than those for which correction values ​​have been derived up to this point by linear interpolation (step S138). This means that correction values ​​have been derived for all colors in the color gamut of the video 51.

[0072] The computer device 30 generates a 3DLUT using the correction values ​​derived by search and interpolation up to step S138 (step S140). Figure 19 is an image diagram of 3DLUT generation. For example, the area enclosed by the dashed line may be generated as a 3DLUT. The number of grid points of the 3DLUT can be determined in advance, for example, by accepting user input. The computer device 30 stores the 3DLUT generated in step S140 in a storage area such as a hard disk, and performs color adjustment processing when displaying images 51 such as background images on the display device 10 in a shooting scene such as a virtual production. More specifically, when outputting images 51 as images 52 to the display device 10, the RGB values ​​as output values ​​of images 52 are replaced with correction values ​​according to the 3DLUT and output to the display device 10.

[0073] (Functional Block Diagram) Figure 20 shows an example of a functional block diagram of the computer device 30. The computer device 30 may be configured to include a correction value derivation unit 310, a correction value holding unit 320, a color adjustment processing unit 330, and an algorithm 350.

[0074] The computer device 30 uses the color information of the first image (image 51) and the color information of the third image (image 54 (or image 53 captured by the imaging device 20 so as to have the same color gamut as image 51)), which is a captured image taken by the imaging device of the second image (image 52) which is the displayed image when the first image is shown on the display device, to perform a color adjustment process on the first image when the first image is shown on the display device.

[0075] The correction value derivation unit 310 derives correction values ​​for color adjustment processing such that the hue of the first image matches the hue of the third image. The correction value derivation unit 310 also derives correction values ​​for multiple colors present within the color gamut of the first image according to a specific algorithm 350, and then uses the derived correction values ​​to derive correction values ​​for colors other than the multiple colors present within the color gamut of the first image by interpolation. The specific algorithm 350 includes a process of searching for color information of the second image in which the hue of the third image matches the hue of the first image.

[0076] Furthermore, the correction value derivation unit 310 may be configured to include a first correction value derivation unit 3110, a second correction value derivation unit 3120, a third correction value derivation unit 3130, and a fourth correction value derivation unit 3140.

[0077] The first correction value derivation unit 3110 derives first correction values ​​for six primary colors, including the three primary colors of light (R, G, B) and the three primary colors of pigment (Y, C, M). The second correction value derivation unit 3120 uses the first correction values ​​derived in the first correction value derivation unit 3110 to derive second correction values ​​for colors that are in the same positions as the six primary colors in the triangle representing the color gamut on the xy chromaticity diagram. The third correction value derivation unit 3130 uses at least the second correction values ​​derived in the second correction value derivation unit 3120 to derive third correction values ​​for colors that are on the sides of the triangle representing the color gamut on the xy chromaticity diagram. The fourth correction value derivation unit 3140 uses at least the third correction values ​​derived in the third correction value derivation unit 3130 to derive fourth correction values ​​for colors that are located inside the triangle representing the color gamut on the xy chromaticity diagram.

[0078] The correction value holding unit 320 uses each correction value derived in the correction value derivation unit 310 and holds it, for example, in the format of a 3DLUT.

[0079] The color adjustment processing unit 330 corrects the color information of the second video using the correction values ​​derived by the correction value derivation unit 310.

[0080] Note that the configuration in Figure 20 is just one example, and the functional configuration of the computer device 30 is not limited thereto. The computer device 30 may consist of a single computer device or multiple computer devices. If the computer device 30 consists of multiple computer devices, for example, each of the functions illustrated in Figure 20 may be handled by multiple computer devices, or a single function may be processed by multiple computer devices.

[0081] (Other Embodiments) In the above embodiment, it was assumed that the video 52 would display a single monochrome image. However, when searching for a correction value, multiple monochrome images with changed colors may be displayed on the display device 10 as a single display image, and this may be captured by the imaging device 20. Figure 21 illustrates a case in which multiple monochrome images are used as a single display image. For example, (1) illustrates a case in which a monochrome image (video 52) with an RGB value of (0,255,0) is displayed as a single display image. (2) illustrates a case in which four monochrome images with RGB values ​​of (0,255,0), (0,255,10), (0,255,20), and (0,255,30), obtained by increasing the B value by 10 each time, are displayed as a single display image.

[0082] In the search in steps 1 to 4, the interval between measurement colors and the interval between changes in the values ​​changed during the search for each measurement color may be different each time the search is conducted.

[0083] In the above embodiment, linear interpolation was used as the interpolation method, but the method is not limited to this. For example, cubic spline interpolation, piecewise cubic Hermitian interpolation, etc., may also be used.

[0084] The color adjustment system according to this embodiment can determine the output value of the image 52 so that the image 54 of the color gamut X captured by the imaging device 20 ultimately matches the hue of the image 51, using the color adjustment method described above. The computer device 30 then transmits a control signal to the display device 10 to execute color adjustment processing when the image 52 is displayed, based on the determined output value of the image 52. The color adjustment system according to this embodiment is useful, for example, in the shooting of a virtual production. The color adjustment system according to this embodiment adjusts only the color of the image 52 when the background material 51 is displayed, rather than adjusting the color of the captured image 53 as in the conventional method. This makes it possible to avoid unnatural colors in the faces and clothing of performers, etc., that are shot together with the background image 52 during the shooting of a virtual production.

[0085] Furthermore, the color of the image 51 may change depending on the display characteristics of the display device 10 when it is displayed by the display device 10, and depending on the shooting characteristics of the shooting device 20 when it is captured by the shooting device 20. Also, the display characteristics and shooting characteristics differ depending on the models of the display device 10 and the shooting device 20. However, according to the color adjustment system of this embodiment, it is possible to make the hues of the image 51 and the image 54 match regardless of the display characteristics and shooting characteristics.

[0086] Furthermore, the color adjustment system according to this embodiment searches for correction values ​​at certain intervals and compensates for other colors by linear interpolation. This method makes it possible to reduce the computational load on the computer device 30 while maintaining a certain degree of accuracy. In addition, the effects of noise are suppressed.

[0087] Although one embodiment of the present invention has been described so far, it goes without saying that the present invention is not limited to the above-described embodiments and may be implemented in various different forms within the scope of its technical concept.

[0088] Furthermore, the scope of the present invention is not limited to the illustrative and described exemplary embodiments, but also includes all embodiments that produce effects equivalent to those aimed at by the present invention. Moreover, the scope of the present invention is not limited to the combination of features defined by each claim, but can be defined by any desired combination of specific features from all disclosed features.

[0089] 1...Color adjustment system 5...Performer 10...Display device 20...Imaging device 30...Computer device 31...Processor 32...RAM 33...ROM 34...Hard disk drive 35...Removable memory 36...Input / output user interface 37...Communication interface 38...Display 51, 52, 53, 54...Image 310...Correction value derivation unit 3110...First correction value derivation unit 3120...Second correction value derivation unit 3130...Third correction value derivation unit 3140...Fourth correction value derivation unit 320...Correction value holding unit 330...Color adjustment processing unit 350...Algorithm

Claims

1. A color adjustment method performed by a computer system including a display device, an imaging device, and a computer device, the method comprising the step of performing a color adjustment process on the first image when the first image is displayed on the display device, using the color information of the first image and the color information of the third image, which is a captured image taken by the imaging device of the second image, which is a displayed image when the first image is displayed on the display device.

2. The color adjustment method according to claim 1, wherein the step of performing the color adjustment process includes deriving a correction value for the color adjustment process such that the hue of the first image matches the hue of the third image.

3. The color adjustment method according to claim 2, wherein the step of deriving the correction values ​​involves deriving correction values ​​for a plurality of colors present within the color gamut of the first image according to a specific algorithm, and then using the derived correction values ​​to derive correction values ​​for colors other than the plurality of colors by interpolation.

4. The color adjustment method according to claim 3, wherein the specific algorithm includes searching for color information of the second image in which the hue of the third image matches the hue of the first image.

5. The color adjustment method according to any one of claims 2 to 4, wherein the step of deriving the correction values ​​is to derive first correction values ​​for six primary colors including the three primary colors of light (R, G, B) and the three primary colors of pigment (Y, C, M); to derive second correction values ​​using the first correction values ​​for colors that are in the same position as the six primary colors in the triangle indicating the color gamut on the xy chromaticity diagram; to derive third correction values ​​using at least the second correction values ​​for colors that are on the sides of the triangle indicating the color gamut on the xy chromaticity diagram; and to derive fourth correction values ​​using at least the third correction values ​​for colors that are located inside the triangle indicating the color gamut on the xy chromaticity diagram.

6. The color adjustment method according to any one of claims 2 to 5, wherein the step of performing the color adjustment process includes correcting the color information of the second video using the correction value derived in the step of deriving the correction value.

7. The color adjustment method according to any one of claims 2 to 6, wherein the correction value derived in the step of deriving the correction value is used and stored in the computer device as 3DLUT data.

8. A computer system for performing the color adjustment method according to any one of claims 1 to 7.

9. A program for causing a computer system to execute the color adjustment method described in any one of claims 1 to 7.

10. A computer-readable recording medium for storing a program for causing a computer system to execute the color adjustment method according to any one of claims 1 to 7.

Images