System and method

Abstract

A system for predicting subsurface light scattering of a real object for relighting of a translucent object in a virtual environment, wherein the system includes: a light pattern projector configured to sequentially project each of multiple different fringe light patterns on the real object; a camera configured to capture an image of the real object for each of the multiple different projected fringe light patterns when the respective fringe light pattern is projected on the real object to obtain phase shift profilometry images; and an image processing device including circuitry configured to: input the phase shift profilometry images into a machine learning algorithm, wherein the machine learning algorithm is configured to predict subsurface light scattering from a plurality of image sections of the phase shift profilometry images, wherein the output of the machine learning algorithm corresponds to a predicted light dot response of a part of the real object that is imaged in the respective image section.

Description

[0001] Sony Semiconductor Solutions Corporation et al.

[0002] SYSTEM AND METHOD

[0003] TECHNICAL FIELD

[0004] The present disclosure generally pertains to a system and a method for predicting subsurface light scattering of a real object for photorealistic relighting of a translucent (or sub-surface scattering) object in a virtual environment.

[0005] TECHNICAL BACKGROUND

[0006] Generally, techniques for digitizing a real object in a virtual environment are known.

[0007] The virtual environments typically include one or more virtual light sources and the light reflection and scattering of the digitized object is taken into account when computing the virtual scene to generate a realistic appearance of the real object in the virtual environment.

[0008] In some cases, when the real object has a surface that includes certain type of materials, for example, with translucent characteristics like skin, marble, wax etc, the light may penetrate the surface and scatters inside the object several times.

[0009] The description of such complex scattering function inside an object may be done by bidirectional subsurface reflection distribution function (“BSSRDF”), which may depend on the object’s shape and (surface) material. Additionally, the scattering may depend on the direction of the incident light and the intensity such that the scattering changes with spatial variance over the object’s surface.

[0010] Hence, generating a realistic appearance of digitized objects may be challenging in some cases. Although there exist techniques for digitizing real objects in a virtual environment, it is generally desirable to improve the existing techniques.

[0011] SUMMARY

[0012] According to a first aspect, the disclosure provides a system for predicting subsurface light scattering of a real object for relighting of a translucent object in a virtual environment, comprising:

[0013] a light pattern projector configured to sequentially project each of multiple different fringe light patterns on the real object; Sony Semiconductor Solutions Corporation et al.

[0014] a camera configured to capture an image of the real object for each of the multiple different projected fringe light patterns when the respective fringe light pattern is projected on the real object to obtain phase shift profilometry images; and

[0015] an image processing device including circuitry configured to:

[0016] input the phase shift profilometry images into a machine learning algorithm, wherein the machine learning algorithm is configured to predict subsurface light scattering from a plurality of image sections of the phase shift profilometry images, wherein the output of the machine learning algorithm corresponds to a predicted light dot response of a part of the real object that is imaged in the respective image section.

[0017] According to a second aspect, the disclosure provides a method for predicting subsurface light scattering of a real object for relighting of a translucent object in a virtual environment, comprising:

[0018] sequentially projecting each of multiple different fringe light patterns on the real object; capturing an image of the real object for each of the multiple different projected fringe light patterns when the respective fringe light pattern is projected on the real object to obtain phase shift profilometry images; and

[0019] inputting the phase shift profilometry images into a machine learning algorithm, wherein the machine learning algorithm is configured to predict subsurface light scattering from a plurality of image sections of the phase shift profilometry images, wherein the output of the machine learning algorithm corresponds to a predicted light dot response of a part of the real object that is imaged in the respective image section.

[0020] Further aspects are set forth in the dependent claims, the drawings and the following description.

[0021] BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Embodiments are explained by way of example with respect to the accompanying drawings, in which:

[0023] Fig. 1 schematically illustrates in a block diagram an embodiment of a system for performing phase shift profilometry;

[0024] Fig. 2 schematically illustrates an embodiment of fringe light patterns;

[0025] Fig. 3 schematically illustrates in a block diagram an embodiment of a system for predicting subsurface light scattering of a real object for digitizing the real object in a virtual environment; Sony Semiconductor Solutions Corporation et al.

[0026] Fig. 4 schematically illustrates in a block diagram an embodiment of a system for predicting subsurface light scattering of a real object for digitizing the real object in a virtual environment; Fig. 5 schematically illustrates an embodiment of an image section of phase shift profilometry images and a respective image section of a light dot response image;

[0027] Fig. 6 schematically illustrates in a flow diagram an embodiment of a method;

[0028] Fig. 7 schematically illustrates in a flow diagram an embodiment of a method; and

[0029] Fig. 8 schematically illustrates in a block diagram an embodiment of a multi-purpose computer.

[0030] DETAILED DESCRIPTION OF EMBODIMENTS

[0031] Before a detailed description of the embodiments under reference of Fig. 3 is given, general explanations are made.

[0032] The digitization of a real object in a virtual environment typically includes the measurement of the three-dimensional (“3D”) shape of the real object.

[0033] A known technique for obtaining information about the 3D shape of a real object is phase shift profilometry (“PSP”), which will be discussed in the following under reference of Fig. 1 and Fig. 2 for enhancing the general understanding of the present disclosure.

[0034] Fig. 1 schematically illustrates in a block diagram an embodiment of a system 1 for performing phase shift profilometry, which also applies to other embodiments of the present disclosure. The system 1 includes a light pattern projector 2, a camera 3 and a computer 4 which basically controls the overall process of light projection and image acquisition and performs the image processing to obtain the 3D shape information.

[0035] The light pattern projector 2 sequentially projects multiple different fringe light patterns 5 on a real object 6 and the camera 3 captures an image of the real object 6 for each of the multiple different projected fringe light patterns 5 when the respective fringe light pattern 5 is projected on the real object to obtain phase shift profilometry images 7.

[0036] Typically, at least three different fringe light patterns 5 are used which are different with respect to a spatial phase shift.

[0037] For example, a set of sinusoidal light patterns may be used, which can be described as follows in the projector space:

[0038] >

[0039]

[0040] Sony Semiconductor Solutions Corporation et al.

[0041] where X is the spatial wavelength for all fringe light patterns belonging to a particular spatial frequency / wavelength and can be measured in pixels per cycle. For example, for a unit frequency sinusoidal signal, it is equal to either the projector’s width or height.

[0042] Moreover, m and n are the horizontal and vertical direction in the projector space, and 6, is the " " >

[0043]

[0044] Fig. 2 schematically illustrates an embodiment of fringe light patterns, which is discussed in the following.

[0045] The fringe light patterns (a), (b) and (c) show a sinusoidal intensity modulation along the x-direction (e.g., horizontal direction) in the projector space with a spatial period of 200 projector pixels, while there is no intensity modulation along the vertical direction for projector pixels with the same x-coordinate.

[0046] Referring again to Fig. 1, typically, the projected fringe light pattern 5 appears distorted from the point of view of the camera 3 due to the shape of the real object 6. This distortion of the fringe light pattern 5 is thus indicative for the 3D shape of the real object 6 such that each image pixel -denoted by a pair of image pixel coordinates (x,y) of the image sensor in the camera - observes a spatial phase according to the photometrically calibrated arrangement between the light pattern projector 2 and the camera 3, the projected fringe light pattern 5 and the 3D shape of the real object 6, wherein the spatial phase is indicative for the 3D shape information.

[0047] The observed intensity reflected of the real object 6 at camera or image pixel (x,y) is given by:

[0048] Ii(x,y) = - [Ld(x,y) cos(O(x,y) + 8j) + Ld(x,y) + Lg(x,y)],

[0049] wherein (x,y) denotes the observed spatial phase, Ld(x,y) is the direct component (i.e. the direct reflection which depends on the phase (x,y) of the surface point) and Lg(x,y) is the global component.

[0050] The direct component Ldand the global component Lgare calculated as follows:

[0051] 2

[0052] Lg= ~ (Io + Ii + 12) ■ L ,

[0053]

[0054] Here, Io, I, and I2are the observed intensities at image pixel (x,y) the three different phase shifts 5; as mentioned above (for N = 3 step PSP FPP; FPP stands for “Fringe Pattern Projection”). Sony Semiconductor Solutions Corporation et al.

[0055] In general, in some embodiments, even more fringe light patterns 5 may be used to improve the acquired 3D shape information, for example, the fringe light pattern 5 may be rotated by

[0056] 90 degrees such that three further different fringe light patterns 5 are used, i.e. three for each spatial direction of the intensity modulation of the fringe light pattern 5 (horizontal and vertical). Moreover, the observed phase <b(x,y) refers typically to a wrapped phase which is limited to the range of [0,2K] of the selected frequency / wavelength.

[0057] The observed phase may be unwrapped for the selected frequency / wavelength by using multiple (higher / lower) spatial frequencies / wavelengths and, thus, in some embodiments, further fringe light patterns 5 with different spatial frequencies / wavelengths are used.

[0058] For example, a series of spatial frequencies fj may be used:

[0059]

[0060] Hence, given the unwrapped phase at one spatial frequency j+1, the unwrapping algorithm calculates the next unwrapped phase at j, i.e. the next higher spatial frequency. The unwrapped phase j can then iteratively be calculated by:

[0061]

[0062] wherein NINT (“Nearest Integer”) refers to the rounding to the nearest Integer and it is started by j = F-l with F being the maximum number of spatial frequencies F. Moreover, it is assumed that:

[0063] <

[0064] With this approach, the unwrapped phases for lower spatial frequencies can be obtained and, thus, in some embodiments, multiple different spatial frequencies are used.

[0065] Spatial frequencies or periods may be, for example, T = 6, 8, 10, 16, 32 and 64 image pixels. The captured phase shift profilometry images 7 may thus include, in some embodiments, one captured image for each fringe light pattern 5 of at least three different fringe light patterns 5, wherein the fringe light patterns 5 are different with respect to a spatial phase shift. Further images may be captured for another spatial direction and / or further spatial frequencies.

[0066] Returning to the general explanations, as mentioned in the outset, however, generating a realistic appearance of digitized objects may be challenging in some cases, for example, for objects with translucent surface characteristics. Sony Semiconductor Solutions Corporation et al.

[0067] While the pure phase shift profilometry may provide, as discussed for example under reference of Fig. 1 and Fig. 2 above, information about the 3D shape of the real object, it may not directly provide information about the subsurface light scattering of the real object.

[0068] Current digitization processes, in some cases, may not consider subsurface light scattering to full extent and may try to emulate the real object inside a Tenderer with global subsurface parameter settings.

[0069] It has thus been recognized that the subsurface light scattering of the real object, for example with translucent properties, should be estimated to achieve a realistic appearance when the real object is digitized in the virtual environment.

[0070] It has been recognized that a dedicated image capturing approach and neural network algorithms may be used to predict subsurface light scattering of translucent objects to enhance realistic appearance in a virtual world (virtual environment). The rendering of real objects may be performed based on acquired 3D shape information and the relighting of a corresponding virtual object may be performed based on vision-based predicted subsurface light scattering information.

[0071] Hence, some embodiments pertain to a system for predicting subsurface light scattering of a real object for relighting of a translucent object in a virtual environment, wherein the system includes:

[0072] a light pattern projector configured to sequentially project each of multiple different fringe light patterns on the real object;

[0073] a camera configured to capture an image of the real object for each of the multiple different projected fringe light patterns when the respective fringe light pattern is projected on the real object to obtain phase shift profilometry images; and

[0074] an image processing device including circuitry configured to:

[0075] input the phase shift profilometry images into a machine learning algorithm, wherein the machine learning algorithm is configured to predict subsurface light scattering from a plurality of image sections of the phase shift profilometry images, wherein the output of the machine learning algorithm corresponds to a predicted light dot response of a part of the real object that is imaged in the respective image section.

[0076] The system uses phase shift profilometry for acquiring 3D shape information of the real object and, thus, uses a structured light approach for acquiring 3D shape information of the real object. Sony Semiconductor Solutions Corporation et al.

[0077] The same information obtained from phase shift profilometry is used for predicting subsurface light scattering.

[0078] Moreover, the system uses, in some embodiments, a combination of a fringe light pattern and a light dot pattern to train a machine learning algorithm to predict subsurface light scattering based on the acquired phase shift profilometry images.

[0079] The granularity of the prediction may be determined by the number of image sections and the corresponding number of light dots.

[0080] The system may be operated, for example, with visible light (typically wavelengths ranging from 380 to 780 nanometers) or near-infrared light (typically wavelengths larger than 780 nanometers and lower than 3000 nanometers for example).

[0081] The system may be used in a real environment in which ambient light can be controlled and, for example, switched off such as in a laboratory, a photo studio, a production environment or the like, but the present disclosure is, of course, not limited to such a use case.

[0082] The system may be used in a real environment in which an intensity of ambient light is relatively low compared to the highest intensity of the fringe light patterns, but the present disclosure is, of course, not limited to such a use case.

[0083] The real object may be any kind of physical object, for example, an object with at least partially translucent surface properties, i.e. a translucent object, which may also be referred to as a subsurface scattering object.

[0084] The virtual environment may include, for example, a virtual space in which one or more virtual objects and one or more virtual light sources are arranged.

[0085] A digitization of the real object includes, for example, rendering the real object in the virtual environment to obtain a virtual object based on the acquired 3D shape information and then perform relighting of the virtual object in the virtual environment based on the subsurface light scattering information.

[0086] Thus, in some embodiments, correspond to a system for predicting subsurface light scattering of a real object for digitizing the real object in a virtual environment.

[0087] The light pattern projector may be based, for example, on Digital Light Processing (“DLP”) technology or Liquid Crystal Display (“LCD”) technology. Sony Semiconductor Solutions Corporation et al.

[0088] DLP projectors typically utilize a digital micromirror device (“DMD”) chip which includes a plurality of small mirrors, and each mirror can be tilted independently. The mirrors reflect light towards or away from the light pattern projector’s lens to project a particular light pattern.

[0089] LCD projectors typically deploy liquid crystal panels to modulate light. These panels act as microscopic shutters, opening and closing to control the passage of light, thereby the light pattern projector can project a particular light pattern.

[0090] Other techniques may be used, as long as the light pattern projector is able to project fringe light patterns (e.g., sinusoidal) with certain phase shifts. Moreover, the light pattern projector should also be able to project a light dot pattern with a certain number of light dots in some embodiments.

[0091] As mentioned above, the light pattern projector is configured to sequentially project each of multiple different fringe light patterns on the real object. The multiple different fringe light patterns have also been discussed above under reference of Fig. 1 and Fig. 2.

[0092] Basically, a fringe light pattern may have a periodic spatial intensity modulation in at least one direction (e.g., horizontal or vertical) such as a spatial binary intensity modulation or a spatial triangular intensity modulation or a spatial sinusoidal intensity modulation or the like.

[0093] In some embodiments, each fringe light pattern has a spatial sinusoidal intensity modulation along one spatial direction.

[0094] As also discussed under reference of Fig. 1 and Fig. 2 above, in some embodiments, the multiple different fringe light patterns are different with respect to at least one of a spatial frequency, a spatial phase shift and a spatial direction of the intensity modulation.

[0095] The camera is not particularly limited, but the camera should have an image sensor which is sensitive for the same wavelength(s) which the projected light of the light pattern projector has. The camera may be a multi-color camera or a single-color camera.

[0096] The image processing device may be or may include a computer, a server, an electronic device, a mobile electronic device (e.g., a smartphone, a tablet, a laptop, a virtual reality device, etc.) or the like.

[0097] The image processing device may be or may include a data processing module in the camera or in the light pattern projector. Thus, in some embodiments, the image processing device is not a separate device, but rather integrated in the camera or the light pattern projector. Sony Semiconductor Solutions Corporation et al.

[0098] The circuitry may be or may include one or more processors. A processor may be or may include an application processor, a central processing unit (“CPU”), a graphical processing unit (“GPU”), a digital signal processor (“DSP”), a field-programmable gate array (“FPGA”), an application specific integrated circuit (“ASIC”) etc.

[0099] The circuitry may include one or more memory components. A memory component may be or may include volatile and non-volatile memory such as static random-access memory (“SRAM”), dynamic RAM (“DRAM”), non-volatile RAM (“NVRAM”), read-only memory (“ROM”), programmable ROM (“PROM”), electrically PROM (“EPROM”), electrically erasable PROM (“EEPROM”), flash memory (e.g., NOR flash or NAND flash) etc. A memory component may be or may include one or more registers, caches, main memories, hard disk drives, solid-state drives etc.

[0100] The circuitry may include one or more data bus interfaces.

[0101] The circuitry may include one or more communication interfaces, wherein each communication interface may be configured to communicate, for example, via a local area network (LAN), a wireless local area network (WLAN), a mobile telecommunications system (GSM, UMTS, LTE, NR etc.), Bluetooth, etc.

[0102] The functionality of the circuitry may be implemented by typical electronic components configured to achieve the functionality as described herein. The functionality of the circuitry may be implemented in parts by typical electronic components and in parts by software configured to achieve the functionality as described herein. The functionality of the circuitry may be implemented by software configured to achieve the functionality as described herein. As mentioned above, the image processing device is configured to input the phase shift profilometry images into a machine learning algorithm, wherein the machine learning algorithm is configured to predict subsurface light scattering from a plurality of image sections of the phase shift profilometry images, wherein the output of the machine learning algorithm corresponds to a predicted light dot response of a part of the real object that is imaged in the respective image section.

[0103] In other words, the machine learning algorithm is configured to predict subsurface light scattering for each of the plurality of image sections based on image pixel values of the respective image section of the phase shift profilometry images, wherein the output of the machine learning algorithm corresponds to a predicted light dot response of a part of the real object that is imaged in the respective image section. Sony Semiconductor Solutions Corporation et al.

[0104] In other words, the machine learning algorithm is configured to learn the individual anisotropic pixel footprint response curve which corresponds to the subsurface light scattering at that location on the real object.

[0105] The machine learning algorithm may be a classical machine learning algorithm, wherein a classical machine learning takes as input predetermined features to perform a prediction or classification.

[0106] A classical machine learning algorithm may be, for example, Logistic Regression, Decision Tree, Support Vector Machines, K-Nearest Neighbors, K-Means Clustering or Principal Component Analysis.

[0107] In the case of a classical machine learning algorithm, the phase shift profilometry images may be pre-processed to calculate values for the predetermined features. The predetermined features may be based, for example, on a spatial variance of image pixel values, spatial gradients, maximum and minimum image pixel values, etc. in an image section. The output may then be a vector or array or matrix with the predicted image pixel values of the light dot response for the respective image section.

[0108] In some embodiments, the machine learning algorithm is based on a neural network.

[0109] The neural network includes an input layer, an output layer and one or more hidden layers, wherein the hidden layers may include one or more convolutional layers, fully-connected layers, recurrent layers or the like.

[0110] In some embodiments, the neural network is based on a U-Net convolutional neural network. A U-Net convolutional neural network consists of two main parts: a contracting path and an expansive path, wherein the architecture is symmetric. U-Net convolutional neural networks are typically used in image analysis and image segmentation.

[0111] The contracting path is like a typical CNN (“Convolutional Neural Network”) and involves repeated application of convolutions, each followed by a rectified linear unit (“ReLU”) and a max pooling operation for example. This path reduces the spatial dimensions while increasing the feature dimensions. Of course, other activation functions than ReLU may be used, for example, the activation function may be a swish activation function.

[0112] The expansive path involves upsampling the feature maps and performing convolutions. It may also include, for example, concatenations with corresponding feature maps from the contracting Sony Semiconductor Solutions Corporation et al.

[0113] path, which helps in retaining spatial information. This path increases the spatial dimensions to produce, for instance, a segmented output.

[0114] As mentioned above, the phase shift profilometry images are input into the machine learning algorithm.

[0115] Generally, each image of the phase shift profilometry images is divided into a plurality of image sections and each image section is processed. Each of the plurality of image sections includes a pixel footprint response at that surface point on the real object which is the ground-truth subsurface scattering which is to be learned and approximated using the machine learning algorithm and subsequent eventual neural relighting.

[0116] The phase shift profilometry images may be input as wrapped or unwrapped phase images. The calculation of the unwrapped phase image is generally known and, for instance, also discussed under reference of Fig. 1 above.

[0117] An image section refers to a set of neighboring image pixels indicated by a range of image pixel coordinates, for example, an image may have 1024x1024 pixels and the image may be divided into 64x64 image sections such that each phase shift profilometry image may divided into 16x16 image pixel areas, each corresponding to an image section.

[0118] As mentioned above, the machine learning predicts subsurface light scattering from the plurality of image sections of the phase shift profilometry images, i.e. the machine learning predicts, for each image section of the phase shift profilometry images, the subsurface light scattering.

[0119] Generally, the machine learning outputs a single prediction for each image section for all of the phase shift profilometry images.

[0120] As mentioned above, the output of the machine learning algorithm corresponds to a predicted light dot response of a part of the real object that is imaged in the respective image section. Generally, the output thus corresponds to a prediction of image pixel values for the respective image section, wherein the image pixel values represent a light dot response of a part of the real object that is imaged in the respective image section.

[0121] The predicted light dot response, generated based on the image pixel values of the image section of the phase shift profilometry images, corresponds to a predicted light reflection and light scattering as it might be observed by the camera when the corresponding part of the real object is illuminated with a (small) light dot. Sony Semiconductor Solutions Corporation et al.

[0122] The output may thus be a vector or array or matrix or the like with the predicted image pixel values of the light dot response for the respective image section.

[0123] It has been recognized that, as a light dot may be small on the real object, the surface profile of the real object does not have much influence on the reflection and scattering of the light dot and thus on the shape of the reflected and scattered light dot. But the subsurface light scattering has an influence on the shape of the reflected and scattered light dot.

[0124] Hence, the light dot response is indicative for the subsurface light scattering of a part of the real object that is imaged with an image section.

[0125] Therefore, the machine learning algorithm is configured, i.e. trained, with light dot response images.

[0126] Thus, in some embodiments:

[0127] the light pattern projector is configured to project a light dot pattern on the real object; the camera is configured to capture an image of the real object when the light dot pattern is projected on the real object to obtain a light dot response image; and

[0128] the image processing device is configured to train the machine learning algorithm based on the phase shift profilometry images as training input, the predicted light dot responses for the plurality of image sections as training output and the light dot response image as target output. Accordingly, the system is configured to generate its own target data by projecting light dots on the real object and capturing an image when the light dots are projected.

[0129] Typically, the number of light dots may be adapted to the number of image sections or vice versa or may be predefined. In the example discussed above, the number of light dots would correspond to 64x64 such that there is one light dot projected for each image section.

[0130] Thus, the machine learning algorithm learns how the light dot responses in an image section of the light dot response image is connected with certain phase shift profilometry characteristics in that image section.

[0131] The captured light dot response image includes the actual or ground-truth subsurface light scattering occurring on that surface point of the real object.

[0132] Typically, the more light dots are projected, the more granular the images can be divided into image sections and, thus, the sampling of the spatial subsurface light scattering properties may be finer (the limit may be that the light dot responses should not overlap as this would prevent capturing the subsurface scattering on a surface location of the object). Sony Semiconductor Solutions Corporation et al.

[0133] Moreover, the training may include moving the camera or the light pattern projector to sample different light directions and, thus, obtain information about the subsurface light scattering for various angles of incidence or angles of emergence.

[0134] The training may include comparing the training output, i.e. the predicted light dot responses for the plurality of image sections, with the target data, i.e. the measured light dot response for the respective image section. Based on the comparison, weights updates are generated, for example, by back-propagation of the error.

[0135] In some embodiments, the image processing device is configured to perform denoising of the phase shift profilometry images before input into the machine learning algorithm.

[0136] For example, median filtering may be used. Median filtering is a known technique in image processing used to reduce noise while preserving edges, wherein typically each image pixel value is replaced with the median image pixel value in the neighborhood of the respective image pixel.

[0137] In some embodiments, the image processing device is configured to perform denoising of the light dot response image before using the light response image as target output in a training of the machine learning algorithm.

[0138] In some embodiments, the image processing device is configured to perform demosaicing of the phase shift profilometry images before input into the machine learning algorithm.

[0139] Demosaicing is a known technique in image processing and also known as color reconstruction. The demosaicing includes reconstructing a full color image from incomplete color samples, since the image sensor may be overlaid with a color filter (e.g., Bayer filter) to generate specific color information for each image pixel. The missing color values for each pixel are estimated by interpolating the available data from neighboring image pixels.

[0140] In some embodiments, the demosaicing converts a single channel raw input image into a color image, e.g., an RGB (“Red-Green-Blue”) image.

[0141] In some embodiments, the image processing device is configured to perform demosaicing of the light dot response image before using the light response image as target output in a training of the machine learning algorithm.

[0142] In some embodiments, the image processing device is configured to calculate a global component and a direct component for the phase shift profilometry images and to input the global component and the direct component into the machine learning algorithm. Sony Semiconductor Solutions Corporation et al.

[0143] The calculation of the global component and of the direct component has been discussed under reference of Fig. 1 above. The global component includes some extracted information about the subsurface light scattering in the different image sections such that this information helps the machine learning algorithm to learn the subsurface light scattering prediction, i.e. to predict a light dot response of a part of the real object that is imaged in the respective image section. In particular, the decomposition of a given phase shift profilometry image into its corresponding direct and global components may be used when high-frequency fringe light patterns are used. In some embodiments, the image processing device is configured to digitize the real object in the virtual environment based on the phase shift profilometry images and the predicted subsurface light scattering of the real object.

[0144] In other words, in some embodiments, the image processing device is configured to provide (photorealistic) relighting of the translucent object in the virtual environment based on the phase shift profilometry images and the predicted subsurface light scattering of the real object.

[0145] The phase shift profilometry images allows the image processing device to reconstruct the 3D shape of the real object to generate a virtual object. The predicted subsurface light scattering for each image section allows the image processing device to interpolate subsurface light scattering for the whole virtual object and, thus, enhance a realistic appearance when the virtual object is relighted for different light conditions in the virtual environment.

[0146] Some embodiments pertain to a (corresponding) method for predicting subsurface light scattering of a real object for relighting of a translucent object in a virtual environment, wherein the method includes:

[0147] sequentially projecting each of multiple different fringe light patterns on the real object; capturing an image of the real object for each of the multiple different projected fringe light patterns when the respective fringe light pattern is projected on the real object to obtain phase shift profilometry images; and

[0148] inputting the phase shift profilometry images into a machine learning algorithm, wherein the machine learning algorithm is configured to predict subsurface light scattering from a plurality of image sections of the phase shift profilometry images, wherein the output of the machine learning algorithm corresponds to a predicted light dot response of a part of the real object that is imaged in the respective image section.

[0149] The method may be performed by the system as described herein. Sony Semiconductor Solutions Corporation et al.

[0150] Some embodiments pertain to an image processing device for predicting subsurface light scattering of a real object for digitizing the real object in a virtual environment, wherein the image processing device includes circuitry configured to input phase shift profilometry images of the real object into a machine learning algorithm, wherein the machine learning algorithm is configured to predict subsurface light scattering from a plurality of image sections of the phase shift profilometry images, wherein the output of the machine learning algorithm corresponds to a predicted light dot response of a part of the real object that is imaged in the respective image section.

[0151] The image processing device refers to the image processing device as described herein with respect to the system.

[0152] Some embodiments pertain to an image processing method for predicting subsurface light scattering of a real object for digitizing the real object in a virtual environment, wherein the image processing method includes inputting phase shift profilometry images of the real object into a machine learning algorithm, wherein the machine learning algorithm is configured to predict subsurface light scattering from a plurality of image sections of the phase shift profilometry images, wherein the output of the machine learning algorithm corresponds to a predicted light dot response of a part of the real object that is imaged in the respective image section.

[0153] The image processing method may be performed by the image processing device as described herein.

[0154] The methods as described herein are also implemented in some embodiments as a computer program causing a computer and / or a processor to perform the method, when being carried out on the computer and / or processor. In some embodiments, also a non-transitory computer-readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the methods described herein to be performed.

[0155] The system and methods as described herein may achieve or provide at least one of the following in some embodiments:

[0156] A method and system for neural acquisition and representation of subsurface light scattering. The system includes a projection system and camera system, which are calibrated. Sony Semiconductor Solutions Corporation et al.

[0157] The system includes a projection system used to project stripe and dot patterns in a certain structure. Sinusoidal pattern projection and combination with dot pattern projection is used to acquire information about the object.

[0158] For pattern projection, phase shift profilometry with fringe pattern projection is used, which is an optical metrology technique used to measure the surface profile of an object to obtain high-resolution, three-dimensional surface measurements.

[0159] Only phase shift profilometry images is used as input to a deep learning model to learn individual pixel footprint responses.

[0160] Phase shift profilometry has a high speed and accuracy and it is non-invasive, robust and used for 3D reconstruction and has a high measurement accuracy, high point density and is a low-cost technology.

[0161] The camera pipeline includes pre-processing algorithms.

[0162] A linear projector response curve computation is done and correction using median filtering is performed.

[0163] The pixel footprint responses are estimated at a high granular level.

[0164] Returning to Fig. 3, which schematically illustrates in a block diagram an embodiment of a system 1 for predicting subsurface light scattering of a real object for relighting a translucent object in a virtual environment, which will be discussed in the following.

[0165] The system 1 basically corresponds to the system 1 of Fig. 1 in which the camera 3 captures phase shift profilometry images 7, one for each of multiple different fringe light patterns 5 projected onto the real object 6, which is a translucent object (e.g., a wax object such as a candle).

[0166] Each of the phase shift profilometry images 7 is conceptually divided in a plurality of image sections 8.

[0167] Then, the system 1 projects a light dot pattern 9, as schematically illustrated in Fig. 4, which shows schematically in a block diagram an embodiment of the system 1 in which the system 1 projects the light dot pattern 9 onto the real object 6.

[0168] The number of light dots projected onto the real object 6 corresponds to the number of image sections 8. Sony Semiconductor Solutions Corporation et al.

[0169] When the light dot pattern 9 is projected onto the real object 6, the camera 3 captures an image to obtain a light dot response image 10. Each image section 8 of the light dot response image 10 includes a light dot response 11 of that part of the real object 6 that is imaged in the respective image section 8.

[0170] As the light dots are small, the surface profile of the real object may not have much influence on the reflection and scattering of the light dot and thus on the shape of the reflected and scattered light dot. But the subsurface light scattering has an influence on the shape of the reflected and scattered light dot.

[0171] Hence, the light dot responses 11 are indicative for the subsurface light scattering of the real object 6.

[0172] Fig. 5 schematically illustrates an embodiment of an image section 8a of phase shift profilometry images 7a and a respective image section 8a of a light dot response image 10a, which is discussed in the following.

[0173] The six image sections 8a on the left show an image section of phase shift profile images 7a captured for each of three horizontal sinusoidal fringe light patterns (Il_h, I2_h, I3_h) which are different with respect to the spatial phase and for each of three vertical sinusoidal fringe light patterns (Il_v, I2_v, I3_v) which are different with respect to the spatial phase.

[0174] The image section 8a on the right is the same image section 8a as on the left but it is an image section of a light dot response image 10a captured for a light dot pattern. The image section shows the light dot response 1 la of the part of the real object that is imaged in the image section 8a.

[0175] Fig. 6 schematically illustrates in a flow diagram an embodiment of a method 90, which is discussed in the following.

[0176] The method 90 is performed by the system 1 of Fig. 3 and Fig. 4.

[0177] At 91, phase shift profilometry images of a real object are captured.

[0178] At 92, the direct and the global component (see also discussion under reference of Fig. 1 above) are calculated for each phase shift profilometry image. Moreover, at 92, phase unwrapping is performed.

[0179] At 93, an image is captured when a light dot pattern is projected onto the real object which is used as target data for training a deep learning model, wherein the captured image corresponds to Sony Semiconductor Solutions Corporation et al.

[0180] a light dot response image which is used as target data or target output in a training of a deep learning model (in this embodiment after denoising and demosaicing).

[0181] At 94, denoising is performed for the phase shift profilometry images, the direct component, the global component and the light dot response image.

[0182] At 95, demosaicing is performed for the phase shift profilometry images, the direct component, the global component and the light dot response image.

[0183] At 96, the phase shift profilometry images and the direct component and the global component are input into the deep learning model as input, which is also used as training input in a training of the deep learning model, based on which the deep learning model predicts subsurface light scattering for a plurality of image sections, wherein the prediction corresponds to a light dot response in each image section. Moreover, the measured light dot responses, as present in the target data or the target output, are compared with the predicted light dot responses to train the deep learning model, for example, by back-propagating the error.

[0184] At 97, the real object is digitized in a virtual environment based on the phase shift profilometry images and the predicted light dot responses.

[0185] Fig. 7 schematically illustrates in a flow diagram an embodiment of a method 100, which is discussed in the following.

[0186] At 101, each of multiple different fringe light patterns are sequentially projected on a real object, as discussed herein.

[0187] At 102, an image of the real object is captured for each of the multiple different projected fringe light patterns when the respective fringe light pattern is projected on the real object to obtain phase shift profilometry images, as discussed herein.

[0188] At 103, the phase shift profilometry images are input into a machine learning algorithm, wherein the machine learning algorithm is configured to predict subsurface light scattering from a plurality of image sections of the phase shift profilometry images, wherein the output of the machine learning algorithm corresponds to a predicted light dot response of a part of the real object that is imaged in the respective image section, as discussed herein.

[0189] At 104, a light dot pattern is projected on the real object, as discussed herein.

[0190] At 105, an image of the real object is captured when the light dot pattern is projected on the real object to obtain a light dot response image, as discussed herein. Sony Semiconductor Solutions Corporation et al.

[0191] At 106, the machine learning algorithm is trained based on the phase shift profilometry images as training input, the predicted light dot responses for the plurality of image sections as training output and the light dot response image as target output, as discussed herein.

[0192] At 107, the real object is digitized in a virtual environment based on the phase shift profilometry images and the predicted subsurface light scattering of the real object, as discussed herein.

[0193] Fig. 8 schematically illustrates in a block diagram an embodiment of a multi-purpose computer 130, which is discussed in the following.

[0194] The computer 130 can be implemented such that it can basically function as any type of image processing device as described herein. The computer has components 131 to 141, which can form a circuitry, such as any one of the circuitries of the image processing device as described herein.

[0195] Embodiments which use software, firmware, programs or the like for performing the methods as described herein can be installed on computer 130, which is then configured to be suitable for the concrete embodiment.

[0196] The computer 130 has a CPU 131 (Central Processing Unit), which can execute various types of procedures and methods as described herein, for example, in accordance with programs stored in a read-only memory (ROM) 132, stored in a storage 137 and loaded into a random-access memory (RAM) 133, stored on a medium 140 which can be inserted in a respective drive 139, etc.

[0197] The CPU 131, the ROM 132 and the RAM 133 are connected with a bus 141, which in turn is connected to an input / output interface 134. The number of CPUs, memories and storages is only exemplary, and the skilled person will appreciate that the computer 130 can be adapted and configured accordingly for meeting specific requirements which arise, when it functions as an image processing device.

[0198] At the input / output interface 134, several components are connected: an input 135, an output 136, the storage 137, a communication interface 138 and the drive 139, into which a medium 140 (compact disc, digital video disc, compact flash memory, or the like) can be inserted.

[0199] The input 135 can be a pointer device (mouse, graphic table, or the like), a keyboard, a microphone, a camera, a light pattern projector, a touchscreen, a time-of-fight device, etc. Sony Semiconductor Solutions Corporation et al.

[0200] The output 136 can have a camera, a light pattern projector, a display (liquid crystal display, cathode ray tube display, light emittance diode display, etc.), loudspeakers, etc.

[0201] The storage 137 can have a hard disk, a solid-state drive and the like.

[0202] The communication interface 138 can be adapted to communicate, for example, via a local area network (LAN), wireless local area network (WLAN), mobile telecommunications system (GSM, UMTS, LTE, NR etc.), Bluetooth, infrared, etc.

[0203] It should be noted that the description above only pertains to an example configuration of computer 130. Alternative configurations may be implemented with additional or other sensors, storage devices, interfaces or the like. For example, the communication interface 138 may support other radio access technologies than the mentioned UMTS, LTE and NR.

[0204] It should be recognized that the embodiments describe methods with an exemplary ordering of method steps. The specific ordering of method steps is however given for illustrative purposes only and should not be construed as binding.

[0205] All units and entities described in this specification and claimed in the appended claims can, if not stated otherwise, be implemented as integrated circuit logic, for example on a chip, and functionality provided by such units and entities can, if not stated otherwise, be implemented by software.

[0206] In so far as the embodiments of the disclosure described above are implemented, at least in part, using software-controlled data processing apparatus, it will be appreciated that a computer program providing such software control and a transmission, storage or other medium by which such a computer program is provided are envisaged as aspects of the present disclosure.

[0207] Note that the present technology can also be configured as described below.

[0208] (1) A system for predicting subsurface light scattering of a real object for relighting of a translucent object in a virtual environment, wherein the system includes:

[0209] a light pattern projector configured to sequentially project each of multiple different fringe light patterns on the real object;

[0210] a camera configured to capture an image of the real object for each of the multiple different projected fringe light patterns when the respective fringe light pattern is projected on the real object to obtain phase shift profilometry images; and

[0211] an image processing device including circuitry configured to: Sony Semiconductor Solutions Corporation et al.

[0212] input the phase shift profilometry images into a machine learning algorithm, wherein the machine learning algorithm is configured to predict subsurface light scattering from a plurality of image sections of the phase shift profilometry images, wherein the output of the machine learning algorithm corresponds to a predicted light dot response of a part of the real object that is imaged in the respective image section.

[0213] (2) The system of (1), wherein:

[0214] the light pattern projector is configured to project a light dot pattern on the real object; the camera is configured to capture an image of the real object when the light dot pattern is projected on the real object to obtain a light dot response image; and

[0215] the image processing device is configured to train the machine learning algorithm based on the phase shift profilometry images as training input, the predicted light dot responses for the plurality of image sections as training output and the light dot response image as target output. (3) The system of (1) or (2), wherein the image processing device is configured to perform denoising of the phase shift profilometry images before input into the machine learning algorithm.

[0216] (4) The system of any one of (1) to (3), wherein the image processing device is configured to perform demosaicing of the phase shift profilometry images before input into the machine learning algorithm.

[0217] (5) The system of any one of (1) to (4), wherein each fringe light pattern has a spatial sinusoidal intensity modulation along one spatial direction.

[0218] (6) The system of any one of (1) to (5), wherein the multiple different fringe light patterns are different with respect to at least one of a spatial frequency, a spatial phase shift and a spatial direction of the intensity modulation.

[0219] (7) The system of any one of (1) to (6), wherein the image processing device is configured to calculate a global component and a direct component for the phase shift profilometry images and to input the global component and the direct component into the machine learning algorithm. (8) The system of any one of (1) to (7), wherein the image processing device is configured to provide relighting of the translucent object in the virtual environment based on the phase shift profilometry images and the predicted subsurface light scattering of the real object.

[0220] (9) The system of any one of (1) to (8), wherein the machine learning algorithm is based on a neural network. Sony Semiconductor Solutions Corporation et al.

[0221] (10) The system of (9), wherein the neural network is based on a U-net convolutional neural network.

[0222] (11) A method for predicting subsurface light scattering of a real object for relighting of the translucent object in a virtual environment, wherein the method includes:

[0223] sequentially projecting each of multiple different fringe light patterns on the real object; capturing an image of the real object for each of the multiple different projected fringe light patterns when the respective fringe light pattern is projected on the real object to obtain phase shift profilometry images; and

[0224] inputting the phase shift profilometry images into a machine learning algorithm, wherein the machine learning algorithm is configured to predict subsurface light scattering from a plurality of image sections of the phase shift profilometry images, wherein the output of the machine learning algorithm corresponds to a predicted light dot response of a part of the real object that is imaged in the respective image section.

[0225] (12) The method of (11), including:

[0226] projecting a light dot pattern on the real object;

[0227] capturing an image of the real object when the light dot pattern is projected on the real object to obtain a light dot response image; and

[0228] training the machine learning algorithm based on the phase shift profilometry images as training input, the predicted light dot responses for the plurality of image sections as training output and the light dot response image as target output.

[0229] (13) The method of (11) or (12), including denoising of the phase shift profilometry images before input into the machine learning algorithm.

[0230] (14) The method of any one of (11), including performing demosai cing of the phase shift profilometry images before input into the machine learning algorithm.

[0231] (15) The method of any one of (11) to (14), wherein each fringe light pattern has a spatial sinusoidal intensity modulation along one spatial direction.

[0232] (16) The method of any one of (11) to (15), wherein the multiple different fringe light patterns are different with respect to at least one of a spatial frequency, a spatial phase shift and a spatial direction of the intensity modulation.

[0233] (17) The method of any one of (11) to (16), including calculating a global component and a direct component for the phase shift profilometry images and to input the global component and the direct component into the machine learning algorithm. Sony Semiconductor Solutions Corporation et al.

[0234] (18) The method of any one of (11) to (17), including providing relighting of the translucent object in the virtual environment based on the phase shift profilometry images and the predicted subsurface light scattering of the real object.

[0235] (19) The method of any one of (11) to (18), wherein the machine learning algorithm is based on a neural network.

[0236] (20) The method of (19), wherein the neural network is based on a U-net convolutional neural network.

[0237] (21) An image processing device for predicting subsurface light scattering of a real object for digitizing the real object in a virtual environment, wherein the image processing device includes circuitry configured to input phase shift profilometry images of the real object into a machine learning algorithm, wherein the machine learning algorithm is configured to predict subsurface light scattering from a plurality of image sections of the phase shift profilometry images, wherein the output of the machine learning algorithm corresponds to a predicted light dot response of a part of the real object that is imaged in the respective image section.

[0238] (22) The image processing device of (21), wherein the circuitry is configured to train the machine learning algorithm based on the phase shift profilometry images as training input, the predicted light dot responses for the plurality of image sections as training output and an obtained light dot response image as target output.

[0239] (23) The image processing device of (21) or (22), wherein the circuitry is configured to perform denoising of the phase shift profilometry images before input into the machine learning algorithm.

[0240] (24) The image processing device of any one of (21) to (23), wherein the circuitry is configured to perform demosaicing of the phase shift profilometry images before input into the machine learning algorithm.

[0241] (25) The image processing device of any one of (21) to (24), wherein the circuitry is configured to calculate a global component and a direct component for the phase shift profilometry images and to input the global component and the direct component into the machine learning algorithm.

[0242] (26) The image processing device of any one of (21) to (25), wherein the circuitry is configured to digitize the real object in the virtual environment based on the phase shift profilometry images and the predicted subsurface light scattering of the real object. Sony Semiconductor Solutions Corporation et al.

[0243] (27) An image processing method for predicting subsurface light scattering of a real object for digitizing the real object in a virtual environment, wherein the image processing method includes inputting phase shift profilometry images of the real object into a machine learning algorithm, wherein the machine learning algorithm is configured to predict subsurface light scattering from a plurality of image sections of the phase shift profilometry images, wherein the output of the machine learning algorithm corresponds to a predicted light dot response of a part of the real object that is imaged in the respective image section.

[0244] (28) The image processing method of (27), including training the machine learning algorithm based on the phase shift profilometry images as training input, the predicted light dot responses for the plurality of image sections as training output and an obtained light dot response image as target output.

[0245] (29) The image processing method of (27) or (28), including performing median filtering of the phase shift profilometry images before input into the machine learning algorithm.

[0246] (30) The image processing method of any one of (27) to (29), including demosai cing of the phase shift profilometry images before input into the machine learning algorithm.

[0247] (31) The image processing method of any one of (27) to (30), including calculating a global component and a direct component for the phase shift profilometry images and to input the global component and the direct component into the machine learning algorithm.

[0248] (32) The image processing method of any one of (27) to (31), including digitizing the real object in the virtual environment based on the phase shift profilometry images and the predicted subsurface light scattering of the real object.

[0249] (33) A computer program comprising program code causing a computer to perform the method according to any one of (27) to (32), when being carried out on a computer.

[0250] (34) A non-transitory computer-readable recording medium that stores therein a computer program product, which, when executed by a processor, causes the image processing method according to any one of (27) to (32) to be performed.

Claims

Sony Semiconductor Solutions Corporation et al.CLAIMS1. A system for predicting subsurface light scattering of a real object for relighting of a translucent object in a virtual environment, comprising:a light pattern projector configured to sequentially project each of multiple different fringe light patterns on the real object;a camera configured to capture an image of the real object for each of the multiple different projected fringe light patterns when the respective fringe light pattern is projected on the real object to obtain phase shift profilometry images; andan image processing device including circuitry configured to:input the phase shift profilometry images into a machine learning algorithm, wherein the machine learning algorithm is configured to predict subsurface light scattering from a plurality of image sections of the phase shift profilometry images, wherein the output of the machine learning algorithm corresponds to a predicted light dot response of a part of the real object that is imaged in the respective image section.

2. The system of claim 1, wherein:the light pattern projector is configured to project a light dot pattern on the real object; the camera is configured to capture an image of the real object when the light dot pattern is projected on the real object to obtain a light dot response image; andthe image processing device is configured to train the machine learning algorithm based on the phase shift profilometry images as training input, the predicted light dot responses for the plurality of image sections as training output and the light dot response image as target output.

3. The system of claim 1, wherein the image processing device is configured to perform denoising of the phase shift profilometry images before input into the machine learning algorithm.

4. The system of claim 1, wherein the image processing device is configured to perform demosaicing of the phase shift profilometry images before input into the machine learning algorithm.

5. The system of claim 1, wherein each fringe light pattern has a spatial sinusoidal intensity modulation along one spatial direction.

6. The system of claim 1, wherein the multiple different fringe light patterns are different with respect to at least one of a spatial frequency, a spatial phase shift and a spatial direction of the intensity modulation.Sony Semiconductor Solutions Corporation et al.

7. The system of claim 1, wherein the image processing device is configured to calculate a global component and a direct component for the phase shift profilometry images and to input the global component and the direct component into the machine learning algorithm.

8. The system of claim 1, wherein the image processing device is configured to-provide relighting of the translucent object in the virtual environment based on the phase shift profilometry images and the predicted subsurface light scattering of the real object.

9. The system of claim 1, wherein the machine learning algorithm is based on a neural network.

10. The system of claim 9, wherein the neural network is based on a U-Net convolutional neural network.

11. A method for predicting subsurface light scattering of a real object for relighting of a translucent object in a virtual environment, comprising:sequentially projecting each of multiple different fringe light patterns on the real object; capturing an image of the real object for each of the multiple different projected fringe light patterns when the respective fringe light pattern is projected on the real object to obtain phase shift profilometry images; andinputting the phase shift profilometry images into a machine learning algorithm, wherein the machine learning algorithm is configured to predict subsurface light scattering from a plurality of image sections of the phase shift profilometry images, wherein the output of the machine learning algorithm corresponds to a predicted light dot response of a part of the real object that is imaged in the respective image section.

12. The method of claim 11, comprising:projecting a light dot pattern on the real object;capturing an image of the real object when the light dot pattern is projected on the real object to obtain a light dot response image; andtraining the machine learning algorithm based on the phase shift profilometry images as training input, the predicted light dot responses for the plurality of image sections as training output and the light dot response image as target output.

13. The method of claim 11, comprising performing denoising of the phase shift profilometry images before input into the machine learning algorithm.

14. The method of claim 11, comprising performing demosaicing of the phase shift profilometry images before input into the machine learning algorithm.Sony Semiconductor Solutions Corporation et al.

15. The method of claim 11, wherein each fringe light pattern has a spatial sinusoidal intensity modulation along one spatial direction.

16. The method of claim 11, wherein the multiple different fringe light patterns are different with respect to at least one of a spatial frequency, a spatial phase shift and a spatial direction of the intensity modulation.

17. The method of claim 11, comprising calculating a global component and a direct component for the phase shift profilometry images and to input the global component and the direct component into the machine learning algorithm.

18. The method of claim 11, comprising providing relighting of the translucent object in the virtual environment based on the phase shift profilometry images and the predicted subsurface light scattering of the real object.

19. The method of claim 11, wherein the machine learning algorithm is based on a neural network.

20. The method of claim 19, wherein the neural network is based on a U-Net convolutional neural network.

Images