Information processing system, information processing method, and program

Abstract

This information processing system comprises a luminance distribution measurement unit, a subject mask extraction unit, and an image synthesis unit. The luminance distribution measurement unit acquires the luminance distribution of invisible light from an invisible light image of a subject. The subject mask extraction unit acquires the alpha value distribution of the subject on the basis of the luminance distribution of the invisible light. The image synthesis unit synthesizes, with a visible light image of the subject, a CG image indicating a degree of transparency according to the alpha value.

Description

Information processing systems, information processing methods, and programs

[0001] This invention relates to an information processing system, an information processing method, and a program.

[0002] In VFX (Visual Effects), chroma keying is still a common technique for compositing background footage. Chroma keying is a method of compositing another image by making a specific color in a filmed video transparent. In chroma keying, a green screen or blue screen is used as a background screen to extract the area of ​​the person in the foreground. Recognizing the person in the foreground from color information and cutting out that area is called mask extraction.

[0003] Japanese Patent Publication No. 11-004381 ​​Public Relations

[0004] When shooting with a green screen or blue screen, it is not possible to properly composite images of transparent objects (transparent or semi-transparent objects). For example, with highly transparent objects, the color of the background screen is visible through them, resulting in the same masking treatment as the background screen. Conversely, with less transparent objects, the color of the background screen that is visible through them becomes fainter, resulting in the same masking treatment as for opaque objects. Therefore, it is not possible to obtain composite images in which the background is visible through the object.

[0005] Therefore, this disclosure proposes an information processing system, information processing method, and program capable of performing image synthesis that reflects the transparency of the subject.

[0006] According to this disclosure, an information processing system is provided, comprising: a luminance distribution measurement unit that acquires the luminance distribution of invisible light from an invisible light image of a subject; a subject mask extraction unit that acquires the distribution of alpha values ​​of the subject based on the luminance distribution of invisible light; and an image synthesis unit that synthesizes a CG image showing the degree of transparency according to the alpha values ​​with a visible light image of the subject. Furthermore, according to this disclosure, an information processing method is provided in which the information processing of the information processing system is performed by a computer, and a program is provided to enable the computer to perform the information processing of the information processing system.

[0007] This figure shows an example of the configuration of the video production system disclosed. This figure shows an example of the camera configuration. This figure shows an example of a camera equipped with pixels for visible light detection and pixels for invisible light detection. This figure shows an example of a camera acquisition image and a subject mask. This figure shows a magnified view of the area around the bangs of the subject. This figure shows an example of a tracking marker using invisible light. This figure shows an example of the processing flow in camera tracking. This figure explains the representation of transmittance during image synthesis. This figure explains the resolution of the subject mask. This figure explains the shadows that occur during mask extraction. This figure explains a method for correcting the alpha value using a 2D threshold map. This figure explains a method for correcting the alpha value using a 2D threshold map. This figure shows an example of a method for checking the illumination status of invisible light. This figure shows a modified example of the method for acquiring invisible light images. This is a hardware configuration diagram showing an example of a computer that realizes the functions of the video production system.

[0008] Embodiments of the present disclosure will be described in detail below with reference to the drawings. In each of the following embodiments, the same parts will be denoted by the same reference numerals, and redundant descriptions will be omitted.

[0009] The explanation will proceed in the following order: [1. Video Production System Configuration] [2. Camera Configuration] [3. Example of Camera Acquired Image and Subject Mask] [4. Tracking Marker Using Invisible Light] [5. Effect of Extracting Subject Mask Using Invisible Light] [6. Alpha Value Correction Using 2D Threshold Map] [7. Method for Confirming the Illumination Status of Invisible Light] [8. Modified Method for Acquiring Invisible Light Images] [9. Hardware Configuration Example]

[0010] [1. Configuration of the Video Production System] Figure 1 is a diagram showing an example of the configuration of the video production system 1 of this disclosure.

[0011] The video production system 1 is an information processing system that produces videos using VFX. For example, the video production system 1 includes an information processing device 10, a camera 20, an invisible light illumination device 30, a back screen 40, and a visible light illumination device 50.

[0012] The invisible light illumination device 30 emits invisible light L Ｎis an illumination device using the same. The visible light illumination device 50 emits visible light L Ｖ is an illumination device using the same. The invisible light L Ｎ refers to light other than visible light L, such as infrared (IR) and ultraviolet (UV) Ｖ . In the present disclosure, IR is used as the invisible light L Ｎ . The camera 20 captures an invisible light image IM Ｎ and a visible light image IM Ｖ .

[0013] The rear screen 40 is installed behind the subject SB (on the side opposite to the camera 20 side). The invisible light illumination device 30 directs the invisible light L toward the rear screen 40 Ｎ . The subject SB is illuminated from behind by the invisible light L reflected by the rear screen 40 Ｎ . When the rear screen 40 includes an element capable of emitting the invisible light L Ｎ , the rear screen 40 itself can function as the invisible light illumination device 30.

[0014] The information processing device 10 performs subject extraction processing and background video compositing processing. For example, the information processing device 10 includes a luminance distribution measurement unit 11, a subject mask extraction unit 12, an image compositing unit 13, a rendering unit 14, an invisible light illumination control unit 15, and a storage unit 16.

[0015] The camera 20 captures the image of the subject SB illuminated from behind by the invisible light L Ｎ as the invisible light image IM Ｎ . The invisible light image IM Ｎ is an image like a shadow picture in which the image region of the subject SB (subject region SA: see FIG. 4) is a shadow. The luminance distribution measurement unit 11 acquires the invisible light image IM of the subject SB Ｎ to obtain the luminance distribution BD of the invisible light L Ｎ .

[0016] The subject mask extraction unit 12 uses the invisible light image IM of the subject SB Ｎ to obtain the invisible light L ＮBased on the luminance distribution, the distribution of alpha values ​​(transmittance) of the subject SB is obtained. The storage unit 16 stores information regarding the position of the subject area SA, the distribution of alpha values, and the camera angle (position and orientation of the camera 20) as a subject mask SM.

[0017] The rendering unit 14 performs rendering based on the camera angle information to create a CG image IM that will be the background of the subject SB. Ｒ The image synthesis unit 13 generates a visible light image IM of the subject SB from the camera 20. Ｖ The image synthesis unit 13 obtains the visible light image IM of the subject SB. Ｖ and background CG image IM Ｒ and are combined to create a composite image IM Ｓ (See Figure 8)

[0018] Visible light image IM Ｖ and CG image IM Ｒ The blending ratio is determined based on the alpha value. For example, if the alpha value is 50%, the visible light image IM Ｖ and CG image IM Ｒ These are combined in a 50 / 50 ratio. The image synthesis unit 13 performs this synthesis process to produce a CG image IM that shows transparency according to the alpha value. Ｒ The visible light image IM of subject SB Ｖ It can be combined with this.

[0019] The invisible light illumination control unit 15 controls the invisible light illumination device 30 to emit invisible light L Ｎ The lights are turned on and off. The rear screen 40 emits invisible light L Ｎ If the light-emitting element is present, the invisible light illumination control unit 15 will generate an invisible light image IM Ｎ The on / off states of each light-emitting element can be controlled to coincide with the timing of the photo shoot.

[0020] [2. Camera Configuration] Figure 2 shows an example of the configuration of camera 20.

[0021] The camera 20A in Figure 2 includes a visible light camera 21, an invisible light camera 22, a spectral mirror 23, and a lens optical system 24. Light incident through the lens optical system 24 is filtered by the spectral mirror 23 (e.g., a cold mirror) to produce visible light L. Ｖ and invisible light L Ｎ It is spectrally separated. The visible light camera 21 uses the visible light L reflected by the spectral mirror 23. Ｖ The invisible light camera 22 detects invisible light L transmitted through the spectral mirror 23. Ｎ It detects the following. If the spectral analysis is insufficient, a visible light cut filter may be placed in front of the invisible light camera 22.

[0022] Camera 20A is a coaxial camera in which the optical axis of the visible light camera 21 and the optical axis of the invisible light camera 22 are aligned. The visible light camera 21 captures a visible light image IM Ｖ The invisible light camera 22 takes an invisible light image IM. Ｎ The system captures images. The visible light camera 21 and the invisible light camera 22 are controlled by a synchronization signal and operate at the same shutter speed. Since the optical axes of the visible light camera 21 and the invisible light camera 22 are coaxial, no parallax occurs. Therefore, the subject SB can be moved in the depth direction.

[0023] Furthermore, if camera 20 is not moved in the depth direction, the visible light camera 21 and the invisible light camera 22 can be arranged side by side perpendicular to the optical axis, rather than in a coaxial configuration. The size and resolution of the visible light camera 21 and the invisible light camera 22 do not necessarily have to match.

[0024] In the example in Figure 2, two cameras, a visible light camera 21 and an invisible light camera 22, are used. However, a single camera can also be used, with pixels for visible light detection and pixels for invisible light detection mounted on the same substrate. Figure 3 shows the visible light detection pixel PX Ｒ PX Ｇ PX Ｂ and PX pixels for invisible light detection Ｎ This figure shows an example of a camera 20B equipped with PX pixels. Ｒ , pixel PX Ｇ and pixel PX ＢThese devices detect red light, green light, and blue light, respectively.

[0025] [3. An example of a camera-acquired image and subject mask] Figure 4 shows an example of a camera-acquired image and subject mask SM.

[0026] The subject SB is invisible light L reflected by the rear screen 40. Ｎ It is illuminated from behind by [the light source]. The left side of Figure 4 shows the invisible light image IM of the subject SB. Ｎ This shows that the subject SB is invisible light L Ｎ It is illuminated from behind by the luminance distribution measurement unit 11. Ｎ The area obscured by the subject SB is acquired as the subject area SA to be keyed.

[0027] The central part of Figure 4 is the invisible light image IM. Ｎ The subject mask SM obtained from the source is shown. For example, the subject mask SM defines the alpha value of the subject region SA as 10% and the alpha value of the region outside the subject region SA as 0%.

[0028] The right side of Figure 4 shows the visible light image IM of subject SB. Ｖ This shows that the subject SB is illuminated by visible light L from the visible light illumination device 50. Ｖ It is illuminated from the front (camera 20 side) by the image synthesis unit 13. Ｖ The texture of the subject SB is obtained, and the background CG image IM is composited at a ratio corresponding to the alpha value. Ｒ Combine with it.

[0029] Figure 5 is a magnified view of the area around the bangs of the person who will be the subject SB. As shown on the right side of Figure 5, the subject mask SM is the visible light image IM Ｖ Fine details such as hair and the shape of edges such as glasses are faithfully reproduced. In conventional mask extraction using chroma key, the color of the background screen (green or blue) is reflected in the gaps of the hair, making areas that should be the subject area (SA) transparent. On the other hand, in the method disclosed herein, there is no influence from color reflection, so the shapes of fine details and edges are reproduced with high accuracy.

[0030] [4. Tracking marker using invisible light] Figure 6 shows invisible light L Ｎ This figure shows an example of a tracking marker MK using [a specific technology / tool].

[0031] The information processing device 10 has a marker display unit 17 and a tracking unit 18. The marker display unit 17 displays invisible light L, which becomes the tracking marker MK, on ​​the rear screen 40. Ｎ The pattern is displayed. The tracking marker MK is a marker used to detect the camera angle. The tracking marker MK is displayed on the rear screen 40. The tracking unit 18 displays an invisible light image IM Ｎ The camera angle is detected based on the shape of the tracking marker MK that is captured in the image. Ｎ By displaying it in this way, the tracking marker MK is visible light image IM Ｖ It will no longer affect the filming.

[0032] As a method for detecting the camera angle, known methods such as PnP (Perspective-n-Point) can be used. For example, the tracking unit 18 uses an invisible light image IM Ｎ The tracking unit 18 extracts feature points of the tracking marker MK from the known feature points of the tracking marker MK and the invisible light image IM. Ｎ The tracking unit 18 detects the correspondence between the feature points extracted from the data. The tracking unit 18 uses the PnP method to estimate the camera angle from the positional relationship between the corresponding feature points.

[0033] The method of displaying the tracking marker MK is not particularly limited. For example, the marker display unit 17 can control a projector to project the tracking marker MK onto the rear screen 40. The rear screen 40 is made of invisible light L Ｎ If the system includes multiple light-emitting elements capable of emitting light, the marker display unit 17 can control the lighting and extinguishing of each light-emitting element to display the tracking marker MK on the rear screen 40.

[0034] The marker display unit 17 controls the brightness of the tracking marker MK against the ambient invisible light LＮ The brightness of the tracking marker MK is made different from that of the surrounding invisible light L. This makes it possible to make the tracking marker MK appear on the back screen 40. The brightness of the tracking marker MK is different from that of the surrounding invisible light L. Ｎ The brightness can be greater or less than the specified value.

[0035] For example, the rear screen 40 emits invisible light L Ｎ When a large number of light-emitting elements capable of emitting light are provided, the brightness of the tracking marker MK can be made higher than its surroundings by selectively lighting the light-emitting elements at the positions corresponding to the tracking marker MK. Conversely, the brightness of the tracking marker MK can be made lower than its surroundings by selectively turning off the light-emitting elements at the positions corresponding to the tracking marker MK.

[0036] The marker display unit 17 can move the display position of the tracking marker MK to a position where the display is not obstructed by the subject SB, in accordance with the movement of the subject SB. Figure 6 shows how the display position of the tracking marker MK shifts upward or downward when the field of view AV of the camera 20 moves upward or downward. In this case, the marker display unit 17 controls the display position of the tracking marker MK so that it does not interfere with the subject SB. With this configuration, the tracking marker MK is not obstructed by the subject SB. Therefore, tracking is less likely to be lost.

[0037] The marker display unit 17 can control the display timing of the tracking marker MK so that it does not affect mask extraction. The tracking marker MK becomes a problem when the invisible light image IM of the subject SB is affected. Ｎ From invisible light L Ｎ This occurs during the process in which the luminance distribution is acquired (distribution acquisition process). Therefore, the marker display unit 17 can display the tracking marker MK at a timing different from that of the distribution acquisition process.

[0038] For example, when performing mask extraction at a high frame rate of 120 fps, even-numbered frames can be assigned to mask extraction, and odd-numbered frames can be assigned to displaying and capturing tracking markers (MK). With this configuration, the tracking markers (MK) will not affect the distribution acquisition process. As a result, a highly accurate subject mask (SM) can be obtained.

[0039] Figure 7 shows an example of the processing flow in camera tracking. In the example in Figure 7, an IR camera that detects IR is used as the invisible light camera 22. An RGB camera with R (red), green (G), and B (blue) pixels is used as the visible light camera 21. A self-emissive display with built-in infrared LEDs is used as the back screen 40.

[0040] The invisible light illumination control unit 15 controls the invisible light illumination device 30 to emit invisible light L Ｎ Turn on the light (step S0). The IR camera uses invisible light L Ｎ In accordance with the timing of the lights turning on, an invisible light image IM Ｎ The invisible light image IM is captured (step S1). The tracking unit 18 captures the invisible light image IM. Ｎ Feature points are extracted from the tracking marker MK captured in the image. The tracking unit 18 estimates the camera angle (position and orientation of the camera 20) from the positional relationship between the extracted feature points and the feature points of the known tracking marker MK (step S2).

[0041] RGB cameras use visible light imaging (IM). Ｖ The camera takes a picture (step S3). The RGB camera acquires lens parameters (step S4). The RGB camera calculates the shooting area and field of view in the virtual space to be rendered based on the camera angle (step S5). The RGB camera captures a visible light image IM. Ｖ Step S6 detects the image area of ​​the subject SB (subject area SA). The RGB camera determines whether or not the subject SB is within the field of view AV of the camera 20 based on the detection result of the subject area SA (Step S7).

[0042] If the subject SB is within the field of view AV (Step S7: Yes), the marker display unit 17 determines the display area of ​​the tracking marker MK as an area within the rear screen 40 that does not interfere with the subject SB (Step S8). The marker display unit 17 turns on or off the infrared LEDs within the determined display area to display the tracking marker MK (Step S9). If the subject SB is not within the field of view AV (Step S7: No), the marker display unit 17 maintains the current display area of ​​the tracking marker MK.

[0043] [5. Effects of extracting subject masks using invisible light] Below, invisible light L Ｎ This explains the effect of extracting the subject mask SM using [a specific method / tool].

[0044] Figure 8 is a diagram illustrating the representation of transmittance during image synthesis. The left side of Figure 8 shows an example of mask extraction using conventional chroma keying (comparative example). The right side of Figure 8 shows an example of mask extraction using the method of this disclosure (example). In both the comparative example and the example, the upper part is the synthesized image IM. Ｓ The lower side shows the subject mask SM.

[0045] The subject is holding a translucent plastic bag. In shooting using a green screen or blue screen (comparative example), the color of the back screen (green, blue) is visible through the plastic, so the entire plastic bag is extracted as the subject mask SM. Therefore, the CG image IM Ｒ When combined, the background CG image IM is created from the vinyl. Ｒ A composite image IM that appears transparent. Ｓ It cannot be obtained.

[0046] On the other hand, the method (example) of this disclosure yields a subject mask SM with a multi-bit transmittance representation corresponding to the alpha value. Therefore, correct image synthesis is performed taking into account the transparency of the subject SB. In the example in Figure 8, a gradient corresponding to the transmittance is generated in the subject mask SM of the part corresponding to the vinyl. Composite image IM of the part with high transmittance Ｓ Now, the background CG image IM Ｒ It's visible through it.

[0047] In back-projection type illumination methods such as those disclosed herein, invisible light L leaks out from the back. Ｎ The amount of light can be used as transmittance. In conventional methods using chroma key, the result of mask extraction is a binary representation of 0 and 1, making it impossible to perform image synthesis that takes transmittance into consideration. In the method disclosed herein, depending on the performance of the camera 20 used for shooting, it is possible to represent transmittance with a high number of bits, such as 10 bits or 16 bits. In addition, invisible light L Ｎ Since the amount of light can be used as transmittance, the subject mask SM is an invisible light image IM Ｎ It can be easily created from this. Therefore, high-resolution composite image IM Ｓ It will also be possible to display this in real time.

[0048] To achieve a more precise representation of transmittance, invisible light L Ｎ The wavelength may be adjusted. For example, invisible light L Ｎ NIR (Near Infrared) can be used as the light source. With this configuration, invisible light L Ｎ This makes it easier for the light source (SB) to penetrate transparent objects such as vinyl. As a result, the accuracy of image synthesis based on the transmittance of the object (SB) is improved.

[0049] Figure 9 illustrates the process of increasing the resolution of a subject mask SM. The left side of Figure 9 shows a subject mask SM with fine lines, such as hair. In conventional mask extraction using chroma key, the resolution of fine lines is low, and the boundary (edge) between the subject SB and the background may appear unnatural. This is thought to be due to the color (green, blue) of the back screen 40 being reflected in the fine lines, such as hair. The method disclosed here eliminates color reflection, thus increasing the resolution of fine lines. The composite image IM is shown on the right side of Figure 9. Ｓ However, the shape of the hair is reproduced with great precision.

[0050] Figure 10 illustrates the shadow SD that occurs during mask extraction. In conventional mask extraction, a shadow SD is created behind the subject SB due to lighting from the front (e.g., lighting from a key light). This shadow SD is reflected in the subject mask SM (see the left side of Figure 10).

[0051] On the other hand, in the mask extraction of the present disclosure, the subject SB is illuminated from the back side, and mask extraction is performed in the manner of a shadow picture. The silhouette of the subject SB is acquired as the subject mask SM. Invisible light L is applied from the front of the subject SB Ｎ Unlike the case of irradiating, the shadow SD of the subject SB does not occur. Therefore, an accurate subject mask SM excluding the influence of the shadow SD can be obtained. The composite image IM Ｓ is also not affected by the shadow SD (see the right side of FIG. 10).

[0052] In addition to the effects described above, the following effects are achieved over the conventional mask extraction using chroma key. First, in the conventional method, the same color (green, blue) as the back screen 40 cannot be used as the color of costumes or props. This imposes restrictions on costume design and prop selection.

[0053] Further, in the conventional method, the color of the back screen may be reflected by the subject SB, resulting in an unnatural color tone. A glossy object such as metal reflects the color of the back screen 40, so the surface of the object may be made transparent. In a fast-moving scene, the back screen 40 may be seen through due to motion blur, making keying difficult.

[0054] Furthermore, when a green back or the like is used, flat, white and uniform lighting is required. However, in image compositing, there is a demand to carefully design the lighting so that the subject SB and the CG image IM Ｒ blend well. For example, in a scene where the setting sun is shining, it is preferable to cast orange light on the subject SB. However, if orange light is applied to the subject SB, a problem arises when mask extraction is performed relying on hue information.

[0055] The method of the present disclosure is advantageous for the problems as described above. In the method of the present disclosure, color-based mask extraction is not performed. Therefore, the above-described problems do not occur. Further, in the method of the present disclosure, the subject mask SM is created using invisible light L Ｎ . Since visible light illumination is not used for mask extraction, mask extraction is possible even in a dark studio.

[0056] ["6.2 Correction of Alpha Value Using 2D Threshold Map"] FIG. 11 and FIG. 12 are diagrams for explaining an alpha value correction method using a 2D threshold map MP.

[0057] As shown in FIG. 4, the invisible light L on the rear screen 40 Ｎ does not necessarily have a uniform luminance distribution. If the light amount of the invisible light L transmitted through the subject SB is directly used as the transmittance, an error may occur in the alpha value. In the example of FIG. 12, a region with a high luminance level ("High") and a region with a low luminance level ("Low") exist in the rear screen 40. If the luminance measurement result is used as it is, the region with the luminance level of "Low" may be incorrectly acquired as the subject mask SM. Therefore, it is conceivable to measure the illumination unevenness of the invisible light L in advance and correct the alpha value based on the measurement result. Ｎ Ｎ

[0058] For example, as preprocessing, the camera 20 (invisible light camera 22) captures an illumination distribution image. The illumination distribution image means an image obtained by photographing the state where the rear screen 40 is irradiated with the invisible light L in the absence of the subject SB by the camera 20 (invisible light camera 22). The illumination distribution image shows the luminance distribution of the invisible light L that illuminates the subject SB from behind. Ｎ Ｎ

[0059] For example, the camera 20 performs imaging so as to scan the entire rear screen 40. The luminance distribution measurement unit 11 acquires the illumination distribution image as the invisible light image IM for calibration. Ｃ The luminance distribution measurement unit 11 measures the luminance distribution (illumination unevenness) of the invisible light L emitted from the rear screen 40 based on the invisible light image IM. The storage unit 16 stores the measured illumination unevenness as a 2D threshold map MP. Ｃ Based on the invisible light image IM, the invisible light L emitted from the rear screen 40 Ｎ The luminance distribution (illumination unevenness) is measured. The storage unit 16 stores the measured illumination unevenness as a 2D threshold map MP.

[0060] ​​​​During real-time processing of VP shooting, the subject mask extraction unit 12 acquires a 2D threshold map MP from the storage unit 16. The subject mask extraction unit 12 performs a correction process on the distribution of alpha values ​​of the subject SB to cancel out the effect of lighting unevenness of the subject SB estimated from the 2D threshold map MP.

[0061] For example, the subject mask extraction unit 12 estimates the position of the field of view AV based on the camera angle information. The subject mask extraction unit 12 uses the 2D threshold map MP to determine the position of invisible light L within the field of view AV. Ｎ The luminance distribution is extracted. The subject mask extraction unit 12 refers to the luminance distribution (uneven lighting) within the field of view AV and corrects the alpha value so that the effect of uneven lighting is reduced. The storage unit 16 stores the corrected alpha value distribution. With this configuration, an accurate distribution of alpha values ​​can be obtained even if there is uneven lighting.

[0062] [7. Method for confirming the illumination status of invisible light] Figure 13 shows invisible light L Ｎ This figure shows an example of a method for checking the lighting status.

[0063] The illuminance and light distribution of the invisible light illumination device 30 are invisible to the human eye. To confirm whether the illumination is uniform, it is necessary to check it using an IR camera. However, since this process is complicated, in this example, the rear screen 40 has been modified to make this check easier to perform.

[0064] In the lower right of Figure 13, the element structure of the back screen 40 is shown in an enlarged view. The IR LED 42 emits invisible light L Ｎ LED 41 is capable of emitting light. RGB LED 43 emits visible light L Ｖ The LED 41 is capable of emitting light. The RGB LED 43 is uniformly arranged across the entire back screen 40, surrounded by the IR LED 42.

[0065] The RGB LED 43 and IR LED 42 are driven by the same drive circuit. If there is a malfunction in the drive circuit and variations occur in the light intensity of the IR LED 42, variations will also occur in the light intensity of the RGB LED 43. Therefore, by visually observing the variations in the light intensity of the RGB LED 43, it is possible to predict the variations in the light intensity of the IR LED 42.

[0066] The rear screen 40 functions as an invisible light source by simultaneously illuminating all IR LEDs 42. The invisible light source can function as an invisible light illumination device that illuminates the subject SB from behind. The rear screen 40 also functions as a visible light source by simultaneously illuminating all RGB LEDs 43. The visible light source can be used to predict the brightness distribution of the invisible light source.

[0067] For example, a user attempting to perform videography (VP) selectively turns on a visible light during the pre-processing stage before VP shooting and visually checks the brightness distribution of the visible light. If the brightness distribution of the visible light is uniform, the user predicts that the brightness distribution of the invisible light will also be uniform. The user then switches from the visible light to the invisible light and starts VP shooting. If the brightness distribution of the visible light is uneven or the light is not on, the user can predict that there is a problem with the invisible light and stop VP shooting.

[0068] [8. Modified Method for Acquiring Invisible Light Images] Figure 14 shows the invisible light image IM Ｎ This figure shows a modified example of the method for obtaining [the value].

[0069] In the example in Figure 1, invisible light L Ｎ A back-projection type mask extraction method was employed, in which the subject SB is illuminated from behind. In contrast, in this example, invisible light L Ｎ A forward-projection type mask extraction method is employed, in which the subject SB is illuminated from the front.

[0070] Camera 20 (invisible light camera 22) receives invisible light L reflected from the subject SB and the rear screen 40. ＮThe camera 20 detects the following: The rear screen 40 is located further away from the subject SB than the camera 20. The intensity of invisible light LN reflected by the rear screen 40 is weaker than the intensity of invisible light LN reflected by the subject SB. Therefore, the invisible light image IM Ｎ Therefore, the subject area SA will be displayed brighter than other areas.

[0071] The subject mask extraction unit 12 extracts invisible light L reflected from the subject SB. Ｎ The brightness of the subject SB and the invisible light L surrounding it. Ｎ Based on the difference in brightness, the subject area SA to be keyed is detected. With this configuration, the subject mask SM can be obtained by lighting from the front of the subject SB.

[0072] invisible light L Ｎ For this purpose, a light source with properties that are easily reflected by the subject SB is preferred. In the subject mask SM, the reproducibility of the shape of the hair is a problem, so light with wavelengths that are easily reflected by the hair is required. For this reason, in this example, invisible light L Ｎ SWIR (Short-Wavelength InfraRed) is used as the imaging method. This allows for more accurate reproduction of the subject's SB shape compared to when NIR is used.

[0073] [9. Hardware Configuration Example] The video production system 1, etc., according to the embodiment of this disclosure described above, is implemented by a computer 1000 having a configuration such as that shown in Figure 15.

[0074] The explanation will be given using video production system 1 as an example. Figure 15 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of video production system 1. Computer 1000 has a processing circuit 1100, RAM 1200, ROM 1300, secondary storage device 1400, communication interface 1500, input / output interface 1600, display unit 1700, camera unit 1800, microphone 1900, and speaker 2000. Each part of computer 1000 is connected by a bus 1050.

[0075] The processing circuit 1100 operates based on a program stored in the ROM 1300 or secondary storage device 1400, and controls each part. For example, the processing circuit 1100 loads the program stored in the ROM 1300 or secondary storage device 1400 into the RAM 1200 and executes processing corresponding to various programs.

[0076] ROM 1300 stores boot programs such as the BIOS (Basic Input Output System) executed by the processing circuit 1100 when the computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0077] The secondary storage device 1400 is a computer-readable recording medium that non-temporarily records programs executed by the processing circuit 1100 and data used by such programs. Specifically, the secondary storage device 1400 is a recording medium that records programs for each process of the video production system 1 according to the embodiment of this disclosure, which is an example of program data 1450.

[0078] The communication interface 1500 is an interface for the computer 1000 to connect to the external network 1550. For example, the processing circuit 1100 can receive data from other devices or transmit data it has generated to other devices via the communication interface 1500.

[0079] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the processing circuit 1100 receives data from input devices such as a microphone 1900 or a touch panel via the input / output interface 1600. The processing circuit 1100 also transmits data to output devices such as a display unit 1700 or a speaker 2000 via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium (media). Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Discs), magneto-optical recording media such as MOs (Magneto-Optical Discs), tape media, magnetic recording media, or semiconductor memory.

[0080] The display unit 1700 is an interface for displaying information processed by the computer 1000. The display unit 1700 is, for example, a liquid crystal display or an organic electroluminescent display (OLED display). Alternatively, the display unit 1700 may be a touch panel display device or an image projection device. In this disclosure, when the rear screen 40 is used as a display capable of displaying a background image, the rear screen 40 can constitute a part of the display unit 1700.

[0081] The camera unit 1800 is an interface for the computer 1000 to capture images. The camera 20 of this disclosure can constitute a part of the camera unit 1800. The microphone 1900 is an interface for the computer 1000 to capture sound. The speaker 2000 is an interface for the computer 1000 to output processed sound. The various parts of the computer 1000 are connected by the bus 1050. Each interface does not necessarily have to be located inside the computer 1000, but may be located outside the computer 1000 via a network or the like. Furthermore, each part constituting the computer 1000 may be controlled by a circuit different from the processing circuit 1100. For example, the display unit 1700 may be controlled not by the processing circuit 1100, but by a circuit dedicated to display processing provided in the display unit 1700.

[0082] For example, when computer 1000 functions as a video production system 1 according to an embodiment of this disclosure, the processing circuit 1100 of computer 1000 functions as various detection and control units (e.g., information processing device 10) included in the video production system 1 by executing a program loaded onto RAM 1200. The secondary storage device 1400 stores the information processing program and various data according to this disclosure. The processing circuit 1100 reads and executes program data 1450 from the secondary storage device 1400, but as another example, these programs may be obtained from other devices via an external network 1550. In other words, the secondary storage device 1400 is not limited to being inside computer 1000, but may be located outside computer 1000. The processing circuit 1100 is an example of an integrated circuit, and CPU, MPU, GPU, APU, ASIC, and FPGA can all be considered integrated circuits.

[0083] Furthermore, the term "system" in this specification may refer to the entire video production system 1, or it may include the video production system 1 and servers or other information processing systems. Also, when referring to a system, it may include a processor, which is a processing circuit.

[0084] Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may also occur.

[0085] [Note] Of the processes described in the embodiments of this disclosure described above, all or part of the processes described as being performed automatically may be performed manually, or all or part of the processes described as being performed manually may be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above document and drawings may be changed at will unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0086] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0087] Furthermore, the embodiments of this disclosure described above can be combined as appropriate in areas that do not contradict the processing content. Also, the steps shown in the sequence diagram or flowchart of this embodiment can be changed in order as appropriate. For example, each step may be processed chronologically, repeatedly, or partially in parallel.

[0088] Of the processes described in the embodiments of this disclosure described above, all or part of the processes described as being performed automatically may be performed manually, or all or part of the processes described as being performed manually may be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above documents and drawings may be changed at will unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0089] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0090] Furthermore, the embodiments of this disclosure described above can be combined as appropriate in areas that do not contradict the processing content. Also, the steps shown in the sequence diagram or flowchart of this embodiment can be changed in order as appropriate. For example, each step may be processed chronologically, repeatedly, or partially in parallel.

[0091] [Note] This technology can also be configured as follows: (1) An information processing system comprising: a luminance distribution measurement unit that acquires the luminance distribution of invisible light from an invisible light image of a subject; a subject mask extraction unit that acquires the distribution of the alpha value of the subject based on the luminance distribution of invisible light; and an image synthesis unit that synthesizes a CG image showing the degree of transparency according to the alpha value with the visible light image of the subject. (2) The information processing system according to (1) above, wherein the luminance distribution measurement unit acquires the region in which the invisible light from a back screen installed behind the subject is blocked by the subject as the subject region to be keyed. (3) The information processing system according to (2) above, wherein the back screen has a marker display unit that displays the pattern of invisible light which serves as a tracking marker. (4) The information processing system according to (3) above, wherein the marker display unit moves the display position of the tracking marker to a position in which the display is not blocked by the subject in accordance with the movement of the subject. (5) The information processing system according to (3) above, wherein the marker display unit displays the tracking marker at a timing different from the timing at which the luminance distribution of the invisible light is obtained from the invisible light image of the subject. (6) The information processing system according to any one of (2) to (5) above, wherein the subject mask extraction unit obtains the luminance distribution of the invisible light emitted from the back screen as a 2D threshold map, and performs a correction process on the distribution of the alpha value of the subject to cancel out the effect of the lighting unevenness of the subject estimated from the 2D threshold map. (7) The information processing system according to any one of (2) to (6) above, wherein the invisible light is NIR (Near Infrared). (8) The information processing system according to (1) above, wherein the subject mask extraction unit detects a subject area to be keyed based on the difference between the luminance of the invisible light reflected by the subject and the luminance of the invisible light around the subject. (9) The information processing system described in (8) above, wherein the invisible light is SWIR (Short-Wavelength InfraRed).(10) A computer-based information processing method comprising: obtaining the luminance distribution of invisible light from an invisible light image of a subject; obtaining the distribution of alpha values ​​of the subject based on the luminance distribution of invisible light; and synthesizing a CG image showing the degree of transparency according to the alpha values ​​with a visible light image of the subject. (11) The information processing method according to (10), further comprising: obtaining the area in which the invisible light from a back screen installed behind the subject is blocked by the subject as a subject area to be keyed. (12) The information processing method according to (11), further comprising: displaying the pattern of invisible light that serves as a tracking marker on the back screen. (13) The information processing method according to (12), further comprising: moving the display position of the tracking marker to a position in which the display is not blocked by the subject in accordance with the movement of the subject. (14) The information processing method according to (12), further comprising displaying the tracking marker at a timing different from the timing in which the luminance distribution of invisible light is obtained from the invisible light image of the subject. (15) The information processing method according to any one of (11) to (14) above, comprising: acquiring the luminance distribution of the invisible light emitted from the back screen as a 2D threshold map; and performing a correction process on the distribution of the alpha value of the subject to cancel out the effect of the uneven lighting of the subject estimated from the 2D threshold map. (16) The information processing method according to any one of (11) to (15) above, wherein the invisible light is NIR (Near Infrared). (17) The information processing method according to (10) above, comprising: detecting a subject area to be keyed based on the difference between the luminance of the invisible light reflected by the subject and the luminance of the invisible light around the subject. (18) The information processing method according to (17) above, wherein the invisible light is SWIR (Short-Wavelength Infrared).(19) A program that enables a computer to obtain the luminance distribution of invisible light from an invisible light image of a subject, obtain the distribution of alpha values ​​of the subject based on the luminance distribution of invisible light, and synthesize a CG image showing the degree of transparency according to the alpha values ​​with a visible light image of the subject.

[0092] 1. Video Production System (Information Processing System) 11. Brightness Distribution Measurement Unit 12. Subject Mask Extraction Unit 13. Image Synthesis Unit 17. Marker Display Unit 40. Rear Screen IM Ｎ Invisible light image IM Ｒ CG image IM Ｖ Visible light image L Ｎ Invisible light MK Tracking marker MP 2D threshold map SA Subject area SB Subject

Claims

1. An information processing system comprising: a luminance distribution measurement unit that acquires the luminance distribution of invisible light from an invisible light image of a subject; a subject mask extraction unit that acquires the distribution of alpha values ​​of the subject based on the luminance distribution of invisible light; and an image synthesis unit that synthesizes a CG image showing the degree of transparency according to the alpha values ​​with a visible light image of the subject.

2. The information processing system according to claim 1, wherein the luminance distribution measurement unit acquires the region in which the invisible light from a back screen installed behind the subject is blocked by the subject as the subject region to be keyed.

3. The information processing system according to claim 2, further comprising a marker display unit on the rear screen that displays the pattern of invisible light which serves as a tracking marker.

4. The information processing system according to claim 3, wherein the marker display unit moves the display position of the tracking marker to a position where the display is not obstructed by the subject, in accordance with the movement of the subject.

5. The information processing system according to claim 3, wherein the marker display unit displays the tracking marker at a timing different from the timing at which the luminance distribution of the invisible light is obtained from the invisible light image of the subject.

6. The information processing system according to claim 2, wherein the subject mask extraction unit acquires the luminance distribution of the invisible light emitted from the back screen as a 2D threshold map, and performs a correction process on the distribution of the alpha values ​​of the subject to cancel out the effect of the uneven lighting of the subject estimated from the 2D threshold map.

7. The information processing system according to claim 2, wherein the invisible light is NIR (Near Infrared).

8. The information processing system according to claim 1, wherein the subject mask extraction unit detects a subject region to be keyed based on the difference between the luminance of the invisible light reflected by the subject and the luminance of the invisible light surrounding the subject.

9. The information processing system according to claim 8, wherein the invisible light is SWIR (Short-Wavelength InfraRed).

10. A computer-based information processing method comprising: obtaining the luminance distribution of invisible light from an invisible light image of a subject; obtaining the distribution of alpha values ​​of the subject based on the luminance distribution of invisible light; and synthesizing a CG image showing the degree of transparency according to the alpha values ​​with a visible light image of the subject.

11. The information processing method according to claim 10, comprising acquiring as a subject area to be keyed an area in which the invisible light from a back screen installed behind the subject is blocked by the subject.

12. The information processing method according to claim 11, further comprising displaying the pattern of invisible light that serves as a tracking marker on the rear screen.

13. The information processing method according to claim 12, comprising moving the display position of the tracking marker to a position where the display is not obstructed by the subject, in accordance with the movement of the subject.

14. The information processing method according to claim 12, further comprising displaying the tracking marker at a timing different from the timing at which the luminance distribution of the invisible light is obtained from the invisible light image of the subject.

15. The information processing method according to claim 11, comprising: acquiring the luminance distribution of the invisible light emitted from the back screen as a 2D threshold map; and performing a correction process on the distribution of the alpha values ​​of the subject to cancel out the effect of the uneven lighting of the subject estimated from the 2D threshold map.

16. The information processing method according to claim 11, wherein the invisible light is NIR (Near Infrared).

17. The information processing method according to claim 10, comprising detecting a subject area to be keyed based on the difference between the luminance of the invisible light reflected by the subject and the luminance of the invisible light surrounding the subject.

18. The information processing method according to claim 17, wherein the invisible light is SWIR (Short-Wavelength InfraRed).

19. A program that enables a computer to perform the following actions: obtain the luminance distribution of invisible light from an invisible light image of a subject; obtain the distribution of alpha values ​​of the subject based on the luminance distribution of invisible light; and synthesize a CG image showing the degree of transparency according to the alpha values ​​with the visible light image of the subject.

Images