Method for generating high-resolution computer-generated hologram from low-resolution image, and apparatus therefor

Abstract

Disclosed are a method for generating a high-resolution computer-generated hologram from a low-resolution image, and an apparatus therefor. A memory of a computing apparatus is loaded with an artificial neural network that generates a computer-generated hologram from an image, a processor drives the artificial neural network loaded into the memory, and an input / output unit outputs a computer-generated hologram acquired by inputting an image into the artificial neural network. The artificial neural network comprises: a feature extraction block comprising a convolutional layer for extracting features from an image; a mapping block comprising a residual block for learning the relationship between the image and a computer-generated hologram on the basis of the features extracted by the feature extraction block; and an up-sampling block for up-scaling an output of the mapping block.

Description

Method and apparatus for generating a high-resolution computer-generated hologram from a low-resolution image

[0001] An embodiment of the present invention relates to a method and apparatus for generating a high-resolution computer-generated hologram (CGH) from a low-resolution image, and more specifically, to a method and apparatus for generating a high-resolution computer-generated hologram using a deep learning-based artificial neural network.

[0002] Holographic displays include technology that uses a spatial light modulator (SLM) and a coherent light source, such as a laser, to form a complex wavefront of a desired light wave and generate a light intensity distribution according to that complex wavefront. Since holographic displays can reproduce light waves emitted from real objects, they are attracting significant interest and research as future 3D displays. To implement a holographic 3D display, computer-generated hologram calculation is essential, and the calculated hologram is displayed on a spatial light modulator to modulate an appropriate light wavefront.

[0003] Hologram generation methods are generally based on the Rayleigh-Sommerfeld diffraction integral and the light propagation equations derived therefrom. CGH algorithms based on this include mesh methods, which assume an object is a set of surfaces; point cloud methods, which assume an object is a set of points; and layer-based methods, which generate holograms by dividing the object perpendicularly to the holographic plane.

[0004] Representative methods for generating CGHs with 3D information include the Gerchberg-Saxton algorithm and the Stochastic Gradient Descent algorithm. These algorithms require an iterative optimization process to generate high-performance CGHs. However, this iterative optimization increases hologram generation time, making real-time CGH generation difficult. In other words, while holograms can be generated through iterative optimization, this method takes several seconds to produce, which is insufficient speed for real-time hologram generation. Furthermore, since optimization must be performed for each separate image, creating real-time holographic video using existing optimization methods is virtually impossible. In particular, generating holograms with 3D information requires separate optimization for each depth.

[0005] To address this, research on high-speed CGH generation using deep learning has recently been actively underway. In deep learning-based CGH generation methods, faster hologram generation speeds suitable for displays of 4K (3840×4320) or 8K (7680×4320) or higher are required.

[0006] However, existing deep learning-based CGH generation methods require high-resolution inputs to generate high-speed, high-resolution CGHs. If high-resolution inputs are unavailable, a two-step approach is required: upscaling a low-resolution input to high resolution before generating a high-resolution CGH. Both the conventional iterative optimization process for CGH generation and deep learning-based high-resolution CGH generation methods require high-resolution RGB images or RGBD (RGB information + depth information) images as network inputs.

[0007] General CNNs used for 3D data analysis, such as image processing, consist of a combination of convolutional layers, pooling layers, non-linear activation functions, and batch normalization. Convolutional layers play the role of transmitting features from the input data to the next layer while preserving spatial information of the input data. Convolutional layers apply a filter of a specific width to the input data by moving it at regular intervals. The convolution operation is not calculated for every pixel, but only for pixels within the filter's receptive area. Pooling layers are operations that reduce the spatial size of the data in the horizontal and vertical directions, and are primarily used to reduce information and improve computational efficiency. Activation functions enhance the model's expressiveness by applying non-linearity to the results calculated in each convolutional layer. Batch normalization normalizes the input data of intermediate layers during the neural network training process. Batch normalization helps improve neural network performance by accelerating training speed and reducing problems such as vanishing gradients and exploding gradients. Batch normalization adjusts the data to stay within an appropriate range by normalizing the data distribution using the mean and variance calculated from each batch.

[0008] The technical problem that the embodiments of the present invention aim to solve is to provide a method and apparatus for generating a high-resolution computer-generated hologram from a low-resolution image using a deep learning-based artificial neural network.

[0009] An example of a device for generating a computer-generated hologram according to an embodiment of the present invention for achieving the above technical problem is a computing device comprising a memory, a processor, and an input / output device, wherein an artificial neural network for generating a computer-generated hologram from an image is loaded in the memory, the processor drives the artificial neural network loaded in the memory, and the input / output unit outputs a computer-generated hologram obtained by inputting an image to the artificial neural network, and the artificial neural network comprises: a feature extraction block including a convolutional layer for extracting features from the image; a mapping block including a residual block for learning the relationship between the image and the computer-generated hologram based on the features extracted by the feature extraction block; and an upsampling block for upscaling the output of the mapping block.

[0010] An example of a method for generating a computer-generated hologram according to an embodiment of the present invention for achieving the above technical problem is a method for generating a high-resolution computer-generated hologram from a low-resolution image using an artificial neural network by a computing device including a memory, a processor, and an input / output device, comprising: a step of inputting a low-resolution image to an artificial neural network including a feature extraction block, a mapping block, and an upsampling block; and a step of obtaining a high-resolution computer-generated hologram with improved resolution from the artificial neural network; wherein the feature extraction block includes a convolutional layer that extracts features from the image, the mapping block includes a residual block that learns the relationship between the image and the computer-generated hologram based on the features extracted by the feature extraction block, and the upsampling block upscales the output of the mapping block.

[0011] According to an embodiment of the present invention, a CGH can be calculated at high speed in a holographic display, and a high-resolution CGH can be generated using a low-resolution image even when a high-resolution image is unavailable. Furthermore, the training process of the artificial neural network consumes little time, and high-resolution holograms for various low-resolution images can be generated using a once-trained artificial neural network without undergoing an individual optimization process for each image.

[0012] FIG. 1 is a drawing illustrating an example of a computer-generated hologram generation device according to an embodiment of the present invention.

[0013] FIG. 2 is a drawing illustrating an example of an artificial neural network according to an embodiment of the present invention.

[0014] FIG. 3 is a diagram illustrating an example of a learning method of an artificial neural network according to an embodiment of the present invention.

[0015] FIG. 4 is a diagram illustrating an example of a method for generating a high-resolution CGH using a trained artificial neural network according to an embodiment of the present invention.

[0016] FIG. 5 is a diagram illustrating the configuration of an example of a CGH generating device according to an embodiment of the present invention, and,

[0017] FIGS. 6 and 7 are drawings illustrating experimental examples of a CGH generation method according to an embodiment of the present invention.

[0018] Hereinafter, a method and apparatus for generating a high-resolution computer-generated hologram according to an embodiment of the present invention will be described in detail with reference to the attached drawings.

[0019] FIG. 1 is a drawing illustrating an example of a computer-generated hologram generation device according to an embodiment of the present invention.

[0020] Referring to FIG. 1, a computer-generated hologram generating device (100) (hereinafter referred to as the "CGH generating device") generates a high-resolution CGH (120) based on a low-resolution image (110). In this embodiment, "low resolution" and "high resolution" indicate a relative difference in resolution size and are not limited to a specific resolution. To indicate that the resolution of the image input to the CGH generating device (100) is lower than the resolution of the CGH output by the CGH generating device (110), the terms "low resolution image" (110) and "high resolution CGH" (120) are used, and each resolution of the low resolution and high resolution may be predefined in various ways according to the embodiment.

[0021] The CGH generator (100) generates a high-resolution CGH (120) from a low-resolution image (110) using a deep learning-based artificial neural network. For example, the CGH generator (100) can generate a high-resolution CGH (120) in real time by utilizing deep learning-based super-resolution technology. The structure of the artificial neural network of this embodiment is shown in FIG. 2. In addition, the learning method of the artificial neural network for this embodiment is shown in FIG. 3.

[0022] FIG. 2 is a diagram illustrating an example of an artificial neural network according to an embodiment of the present invention.

[0023] Referring to FIG. 2, an artificial neural network (200) that generates a high-resolution CGH (260) from a low-resolution image (250) includes a feature extraction block (210), a mapping block, and an up-sampling block (230).

[0024] The feature extraction block (210) is a block that extracts features from a low-resolution image (250). The feature extraction block (210) is composed of a convolutional layer that extracts features by convolutioning the input data of each channel with a kernel. The feature extraction block (210) can directly extract features from the input low-resolution image (250) without performing a separate intermediate processing step. Therefore, the feature extraction block (210) can effectively capture important information required for generating a high-resolution CGH (260) while reducing computational complexity and spatial complexity. The feature extraction block (210) can be implemented as a feature extraction block used in existing CNNs (Convolutional Neural Networks) or EDSRs (Enhanced Deep Super-Resolution Networks). In addition to this, various other types of existing feature extraction blocks may be used in this embodiment.

[0025] In one embodiment, the feature extraction block (210) may be implemented with 4 input channels capable of receiving a color image (RGBD = RGB image + depth information) with depth information and 4×N output channels (N is a natural number greater than or equal to 2) capable of outputting extracted features. For example, the feature extraction block (210) may apply 24 different kernels (i.e., filters) of size 3×3 to each of the 4 input channels to expand the output for each channel to 24 channels. In other words, the feature extraction block (210) may generate 24 feature maps by applying 24 kernels to each input channel. As a result, the feature extraction block (210) may generate a tensor having 24 output channels. However, this is merely one example, and the number of output channels, the size and number of kernels, etc., may be varied in various ways depending on the embodiment and are not limited to this embodiment.

[0026] The mapping block includes a residual block (220) that learns the complex relationship between a low-resolution image (250), which is the input of the artificial neural network (200), and a high-resolution CGH (260), which is the output of the artificial neural network (200). The residual block (220) includes a convolution layer and a non-linear activation function layer (e.g., a ReLU layer). In one embodiment, the residual block (220) may be composed of a 'convolution layer - ReLU layer - convolution layer'. The mapping block may be implemented as a structure in which a plurality of residual blocks (220) are connected in series. The residual block (220) inputs the sum of the input value and the output value to the next layer. Since the residual block (220) itself is a widely known configuration, further explanation thereof is omitted. In addition, various existing residual blocks of different structures may be applied to this embodiment, and the structure is not limited to that of this embodiment.

[0027] The mapping block plays an important role in preserving high-frequency information, which is essential for accurately capturing fine details. To enhance the preservation of high-frequency characteristics, the mapping block of the present embodiment is characterized by omitting the pooling layer and the transposed convolution layer, which reduce or expand the spatial dimension of intermediate tensors in conventional CNNs or EDSRs. Additionally, the mapping block of the present embodiment is characterized by omitting batch normalization used in conventional CNNs, etc., to optimize memory usage and increase adaptability to various value ranges.

[0028] The upsampling block (230) upscales the output of the mapping block. In one embodiment, the upsampling block (230) may be composed of a resize layer and a convolution layer to reduce checkerboard artifacts. The resize layer increases the size (i.e., resolution) of the image using interpolation (e.g., nearest interpolation). The convolution layer applies a kernel of a predefined size according to a predefined upscaling factor. If the kernel size is small, the pixel information used for training is reduced, and if the kernel size is large, the amount of training increases, which has the disadvantage of taking a long time for inference and training. Therefore, in one embodiment, considering the problem of pixel information reduction and the problem of the amount of training, the convolution layer may use a kernel of size 'scaling factor * 2 + 1'. For example, if the upscaling factor is x2, the convolutional layer uses a 5×5 kernel, if the upscaling factor is x3, the convolutional layer uses a 7×7 kernel, and if the upscaling factor is x4, the convolutional layer can use a 9×9 kernel.

[0029] Since the resolution is enhanced in the final step of the upsampling block (230), the efficiency of the entire computation of the artificial neural network can be increased. In another embodiment, to generate a high-resolution complex CGH, the output value can be limited to within the range [-π, π] using a hyperblock tangent activation function (tanh) in the final layer of the upsampling block (230).

[0030] In order to generate a high-resolution CGH (260) from a low-resolution image (250) using the artificial neural network (200) of this embodiment, a prior training process is required. The process of training the artificial neural network is examined again in FIG. 3.

[0031] FIG. 3 is a diagram illustrating an example of a learning method for an artificial neural network according to an embodiment of the present invention.

[0032] Referring to FIGS. 1 to 3 together, training data is prepared that includes at least one training image, at least one target CGH, and at least one target reconstructed image (S300).

[0033] In one embodiment, the CGH generator (100) can generate training data based on a high-resolution image (240). The CGH generator can convert the high-resolution image (240) into a low-resolution image (250) and use the low-resolution image (250) as a training image. In one embodiment, the CGH generator (100) can generate a low-resolution training image by applying bicubic interpolation to the high-resolution image (240). For example, the CGH generator (100) can train the artificial neural network of this embodiment by downscaling a high-resolution image of size 384×384 to 192×192, 128×128, or 96×96 using bicubic interpolation, thereby training the artificial neural network to generate a high-resolution CGH that is improved by 2x (x2), 3x (x3), or 4x (x4), respectively.

[0034] Additionally, the CGH generating device (100) can generate a target CGH (280) from a high-resolution image (240) and transmit the target CGH (280) to a spatial light modulator plane to generate a reconstructed target image (290). If training data is prepared in advance, the process (S300) of the CGH generating device (100) preparing training data can be omitted.

[0035] The CGH generator (100) inputs a training image into the artificial neural network (200) shown in FIG. 2 (S310). The CGH generator (100) obtains a high-resolution CGH (260) through the artificial neural network (200) and transmits the high-resolution CGH (260) to a spatial light modulator plane to generate a reconstructed image (270) (S320). For example, the CGH generator (100) can obtain a reconstructed image (270) that is reproduced in a simulation using a numerical propagation algorithm on the generated high-resolution CGH (260). As another example, the CGH generator (100) can obtain a reconstructed image (270) from the high-resolution CGH (260) through an experiment.

[0036] The CGH generator (100) identifies a first error between the reconstructed image (270) and the target reconstructed image (290) of the training data (S330). In one embodiment, the CGH generator (100) may additionally identify a second error between the high-resolution CGH (260) output by the artificial neural network (200) and the target CGH (280) of the training data. In this case, the CGH generator (100) may define the loss function as the sum of the first error and the second error.

[0037] In another embodiment, the first error may be the sum of the MS-SSIM (multiscale-structural similarity index measure) loss and the L1 loss, and the second error may be the L2 loss. In this case, the loss function may be defined as follows, taking into account the weights of each error.

[0038]

[0039] Here, w1, w2, and w3 are weights representing the reflection ratio of each loss. For example, w1 can be set to 0.84, w2 to 0.16, and w3 to 0.04. The values ​​of each weight can be varied depending on the embodiment.

[0040] The CGH generator (100) trains the artificial neural network (200) so that the magnitude of the loss function is reduced (S340). Since the training process itself, which optimizes the parameters of each layer of the artificial neural network, is a widely known configuration, further explanation thereof is omitted.

[0041] In another embodiment, the CGH generator (100) can prepare verification data along with training data in advance, and then, once the training of the artificial neural network (200) is completed, use the verification data to verify the performance of the artificial neural network (200). For example, the verification of the artificial neural network (200) can be repeated for 100 epochs.

[0042] FIG. 4 is a diagram illustrating an example of a method for generating a high-resolution CGH using a trained artificial neural network according to an embodiment of the present invention.

[0043] Referring to FIG. 4, when the artificial neural network (400) trained by the method of FIG. 3 receives a low-resolution image (410) as input, it generates and outputs a high-resolution CGH (420). For example, when the trained artificial neural network (400) receives an image with a resolution of 384×384 as input, it can generate a CGH with a resolution of 768×768, which has a resolution improved by twofold.

[0044] FIG. 5 is a diagram illustrating the configuration of an example of a CGH generating device according to an embodiment of the present invention.

[0045] Referring to FIG. 5, the CGH generating device (100) includes an artificial neural network (500) and a learning unit (510). If the artificial neural network (500) has already completed learning, the learning unit (510) may be omitted. The CGH generating device (100) may be implemented as a computing device including memory, a processor, and an input / output device. In this case, the artificial neural network (500), etc., may be loaded into memory and then executed by the processor. The input / output unit inputs a low-resolution image to the artificial neural network (500) and outputs a high-resolution CGH obtained from the artificial neural network (500).

[0046] The artificial neural network (500) includes a feature extraction block, a mapping block, and an upsampling block. An example of the artificial neural network (500) is illustrated in FIG. 2. The learning unit (510) trains the artificial neural network (500). An example of the method by which the learning unit (510) trains the artificial neural network (500) is illustrated in FIG. 3.

[0047] FIGS. 6 and 7 are drawings illustrating experimental examples of a CGH generation method according to an embodiment of the present invention.

[0048] Referring to FIG. 6, experimental results are shown for cases where a CGH with a resolution improved by 2x (x2) was generated from an input image using an artificial neural network according to an embodiment of the present invention (top figure) and where a CGH with a resolution improved by 3x (x3) was generated (bottom figure). As for the hologram conditions, laser wavelengths of Red (638 nm), Green (520 nm), and Blue (450 nm) were assumed, and a spatial light modulator having a pixel pitch of 6.4 μm was assumed. The generated hologram was propagated from -3 mm to 3 mm through a numerical propagation algorithm to have 128 depths. In addition, an Adam optimizer was used for training the artificial neural network, and the beta value was set to 0.99 and the learning rate to 0.0001. The artificial neural network was trained for 100 epochs, the batch size was set to 4, and the learning rate was reduced by half every 20 epochs. In the experiment, for phase space optical modulation, it was converted into a phase-only hologram using a double phase amplitude encoding method.

[0049] Referring to Fig. 6, the top right image shows the results of replaying the CGH generated by the method of the present embodiment and comparing the replay results at each focal point in the x2, x3, and x4 models. It can be confirmed that the method of the present embodiment has a higher Peak Signal to Noise Ratio (PNSR) compared to the conventional method. In other words, the method of the present embodiment generates a high-resolution CGH by using a low-resolution image directly as input without interpolation, and it can be confirmed that the replay result of the generated CGH is superior to that of the conventional method. When the CGH generated by the method of the present embodiment is replayed, it can be confirmed that the focus is accurately set at the part corresponding to the calculated depth (white box area).

[0050] Referring to FIG. 7, the results of experimentally reproducing a high-resolution CGH generated as an embodiment of the present invention are shown. When the focus is set to a close distance, the close parts are in sharp focus, but the distant parts are out of focus. Conversely, it can be experimentally confirmed that when the focus is set to a distant distance, the close parts are out of focus.

[0051] While conventional hologram generation methods require high-resolution input images, the method of the present embodiment can generate CGH faster than the conventional method using only low-resolution input. Furthermore, according to the present experimental example, it can be confirmed that the quality of the image reconstructed by the method of the present invention is similar to or superior to that of the conventional method.

[0052] The present invention can also be implemented as computer-readable program code on a computer-readable recording medium. A computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. Additionally, computer-readable recording media can be distributed across networked computer systems, allowing computer-readable code to be stored and executed in a distributed manner.

[0053] The present invention has been described above with reference to its preferred embodiments. Those skilled in the art will understand that the present invention may be embodied in modified forms without departing from the essential characteristics of the invention. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the invention is defined by the claims, not by the foregoing description, and all variations within the scope of the claims should be interpreted as being included in the invention.

Claims

1. In a computing device comprising memory, a processor, and an input / output device, An artificial neural network that generates a computer-generated hologram from an image is loaded into the above memory, and The above processor drives the artificial neural network loaded in the memory, and The above input / output unit outputs a computer-generated hologram obtained by inputting an image into the artificial neural network, and The above artificial neural network is, A feature extraction block including a convolutional layer that extracts features from the above image; A mapping block including a residual block that learns the relationship between the image and the computer-generated hologram based on features extracted by the feature extraction block; and A computing device for generating a computer-generated hologram, characterized by including an upsampling block that upscales the output of the above-mentioned mapping block.

2. In Paragraph 1, The image above is a color image containing depth information, and The above feature extraction block is, A computing device for generating a computer-generated hologram, characterized by including four input channels capable of receiving a color image having the above depth information, and 4*N output channels that output a plurality of feature maps obtained by applying N different kernels (N is a natural number greater than or equal to 2) to each input channel.

3. In Paragraph 2, The size of the above kernels is 3×3, and A computing device for generating a computer-generated hologram, characterized in that the above feature extraction block outputs 24 feature maps generated by applying 24 different kernels to 4 channels of input to 24 channels.

4. In Paragraph 1, The above mapping block includes at least one residual block, and Each residual block includes a first convolutional layer; a ReLU layer; and a second convolutional layer, and A computing device for generating a computer-generated hologram, characterized in that each residual block transmits the input of the first convolutional layer and the sum of the second convolutional layer to the next layer.

5. In claim 1, the upsampling block is, A resize layer that expands the size by performing nearest interpolation; and A computing device for generating a computer-generated hologram, characterized by including a convolutional layer that improves resolution using a kernel of a predefined size.

6. In Paragraph 1, It further includes a learning unit for training the artificial neural network; and A computing device for generating a computer-generated hologram, characterized in that the learning unit reduces the resolution of a high-resolution image to generate a low-resolution learning image, inputs the learning image into the artificial neural network to generate a reconstructed image from a computer-generated hologram obtained therefrom, identifies a first error between the reconstructed image and a predefined target reconstructed image, and trains the artificial neural network to reduce the first error.

7. In Paragraph 6, A computing device for generating a computer-generated hologram, characterized in that the first error above includes MS-SSIM loss and L1 loss between the reconstructed image and the target reconstructed image.

8. In Paragraph 6, It further includes a second error detection unit for detecting a second error between the computer-generated hologram and a predefined target computer-generated hologram. A computing device for generating a computer-generated hologram, characterized in that the training unit trains the artificial neural network based on a loss function including the first error and the second error.

9. A method for generating a high-resolution computer-generated hologram from a low-resolution image using an artificial neural network, wherein a computing device comprising memory, a processor, and an input / output device, A step of inputting a low-resolution image into an artificial neural network including a feature extraction block, a mapping block, and an upsampling block; The method includes the step of obtaining a high-resolution computer-generated hologram with improved resolution from the artificial neural network; The above feature extraction block includes a convolutional layer that extracts features from the image, and The mapping block includes a residual block that learns the relationship between the image and the computer-generated hologram based on features extracted by the feature extraction block, and A method for generating a computer-generated hologram characterized by the upsampling block upscaling the output of the mapping block.

10. In Paragraph 9, The method further includes the step of training the artificial neural network; The above training step is, A step of generating a low-resolution training image by reducing the resolution of a high-resolution image; A step of generating a reconstructed image from a computer-generated hologram obtained by inputting the above-mentioned training image into the above-mentioned artificial neural network; A step of identifying a first error between the above-mentioned reconstructed image and a predefined target reconstructed image; and A method for generating a computer-generated hologram characterized by including the step of training the artificial neural network based on the first error.

11. A computer-readable recording medium storing a computer program for performing the method described in paragraph 9.

Images