System and method for processing light field data using microcell

Abstract

Disclosed are a system and method for processing light field data using a microcell. The light field data processing system comprises: an editing unit that generates edited images by editing original images; a microcell generation unit that generates first microcells by dividing the original images, and generates second microcells by dividing the edited images; and an image configuration unit that generates light field images by combining the first microcells with the second microcells.

Description

Light field data processing system and method using microcells

[0001] The present invention relates to a light field data processing system and method using a microcell.

[0002] Existing light field data processing systems were limited to processing low-resolution data, or faced difficulties in processing high-resolution data due to high computational costs and memory requirements.

[0003] The present invention provides a light field data processing system and method using a microcell.

[0004] To achieve the above-mentioned purpose, a light field data processing system according to one embodiment of the present invention includes: an editing unit that edits original images to generate edited images; a microcell generation unit that divides the original images to generate first microcells and divides the edited images to generate second microcells; and an image composition unit that combines the first microcells and the second microcells to generate light field images.

[0005] A light field data processing system according to another embodiment of the present invention includes a microcell generation unit that generates cells by spatially and temporally dividing images acquired from a plurality of cameras, and generates microcells smaller in size than the cells by rearranging the generated cells. Here, the microcells have angular consistency while maintaining the spatial features of the images.

[0006] A light field data processing method according to one embodiment of the present invention comprises the steps of: editing original images to generate edited images; dividing the original images to generate first microcells; dividing the edited images to generate second microcells; and combining the first microcells and the second microcells to generate light field images.

[0007]

[0008] The light field data processing system and method according to the present invention converts images into microcells and then generates light field images using the microcells. That is, the light field data processing system processes light field data using microcells. As a result, excellent data processing performance can be maintained while saving computational resources. Therefore, high-resolution light field data processing can be performed even on devices in limited environments, and high-resolution data can be processed in real time, so it can be utilized in various application fields such as 3D content production, augmented reality (AR), and virtual reality (VR).

[0009] FIG. 1 is a drawing illustrating a light field data processing system according to one embodiment of the present invention.

[0010] FIG. 2 is a flowchart illustrating a light field data processing process according to an embodiment of the present invention.

[0011] FIG. 3 is a diagram illustrating a cell generation process according to one embodiment of the present invention.

[0012] FIG. 4 is a diagram illustrating a microcell generation process according to one embodiment of the present invention.

[0013] FIG. 5 is a diagram illustrating the process of constructing a light field image according to one embodiment of the present invention.

[0014] As used in this specification, singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as "composed" or "comprising" should not be interpreted as necessarily including all of the various components or steps described in the specification, and should be interpreted as meaning that some of the components or steps may be excluded, or that additional components or steps may be included. Furthermore, terms such as "...part," "module," etc., as used in the specification refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software.

[0015]

[0016] The present invention relates to a light field data processing system and method, which can process high-resolution light field data using a microcell.

[0017] Previously, light fields were generated using large-sized images captured by cameras; consequently, this required high computational costs and memory usage, making it difficult to use such light field data processing technology on devices with limited resources.

[0018] Therefore, we propose a light field data processing system capable of processing high-resolution light field data even on devices with limited resources.

[0019]

[0020] Hereinafter, various embodiments of the present invention will be described in detail with reference to the attached drawings.

[0021] FIG. 1 is a diagram illustrating a light field data processing system according to an embodiment of the present invention, FIG. 2 is a flowchart illustrating a light field data processing process according to an embodiment of the present invention, FIG. 3 is a diagram illustrating a cell generation process according to an embodiment of the present invention, FIG. 4 is a diagram illustrating a microcell generation process according to an embodiment of the present invention, and FIG. 5 is a diagram illustrating a light field image composition process according to an embodiment of the present invention.

[0022] Referring to FIG. 1, the light field data processing system of the present embodiment may include an editing unit (100), a microcell generation unit (102), and an image composition unit (104).

[0023] According to one embodiment, the light field data processing system generates microcells having spatial information and angle information for each of the high-resolution original images captured by cameras, and can generate light field images using said microcells. In this case, since the size of the microcells is small, light field data processing can be performed with low computational cost and memory usage.

[0024]

[0025] Below, we will examine the above light field data processing process in detail.

[0026] Referring to FIGS. 1 and 2, the light field processing system obtains high-resolution original images (LF) from a plurality of cameras taken at different points in time. B ) obtain and original images (LF B ) is edited using the editing department (100) to create edited images (LF W ) can be generated (S200). Here, the original image (LF B ) may be a sub-aperture image (SAI) obtained by covering part of the camera lens and taking a picture.

[0027]

[0028] Here, N represents the number of SAIs.

[0029] Specifically, the original images (LF B You can obtain an edited SAI(I) by editing the ) and perform warp-based propagation on the edited SAI(I) by warping the information to match the movement of the object or the movement of the camera.

[0030] Geometric information of the edited SAI(I) is required for the above-mentioned warp-based propagation, and for this purpose, the editing unit (100) can infer a disparity map (D) of the edited SAI(I) using mono depth estimation (single image depth estimation).

[0031] The editing department (100) can generate N warp images using the disparity map (D) inferred in this way, and the warp images are editing images (LF W ) is. That is, the editing department (100) uses the inferred disparity map (D) to edit images (LF W ) can be generated. At this time, edited images (LF W ) may have distortion, blurring, and discontinuity between the object and the background, so the image composition network described below is required to resolve this.

[0032] Meanwhile, original images (LF B Edit the edited images (LF) by editing ) W As long as it is possible to generate ), the editing method is not limited to the above method and various methods can be used.

[0033] Next, the microcell generation unit (102) uses original images (LF B ) and edited images (LF W Cells (pixels) can be generated by dividing each of the cells, and microcells can be generated by rearranging the generated cells (S202 and S204).

[0034] For the sake of convenience, the process of generating cells and microcells will be explained below using a SAI-formatted image (LF) as an example. Here, it will be assumed that the image (LF) has (H×W) spatial resolution, (U×V) angular resolution, and a C channel.

[0035] According to one embodiment, the microcell generating unit (102) can divide an image (LF) at an angle resolution of (u×v) into Ns pieces to generate small images (cells) having a spatial resolution of (h×w), which can be expressed by the following mathematical formula 2. As a result, pixels having a resolution of (u×v×h×w) can form a single cell.

[0036] In this case, spatial partitioning does not overlap, whereas angular partitioning can overlap. Therefore, N in Equation 2 a can be greater than the value obtained by dividing (u×v) by (U×V).

[0037] For example, the microcell generation unit (102) can generate cells having a (4×4) angle resolution by dividing an image (LF) having a (7×7) angle resolution as shown in FIG. 3.

[0038]

[0039] That is, the microcell generation unit (102) can generate cells having low spatial resolution and angular resolution by sequentially performing angular division and spatial division on an image (LF) having high resolution. At this time, the cell may contain sufficient information about spatially and angularly adjacent pixels, even though the size is small.

[0040] According to one embodiment, the microcell generation unit (102) can generate low-resolution cells by angularly dividing an image (LF) having high resolution and then spatially dividing it.

[0041] Then, the microcell generation unit (102) can rearrange each of the cells to generate microcells having a size smaller than the cell. For example, as shown in FIG. 4, pixels requiring angular consistency can be gathered from different images (LF, SAI). That is, the microcell can contain angular information of the images (LF) and can not lose spatial features.

[0042] Due to the characteristics of these microcells, the microcells can achieve angular consistency while maintaining spatial features.

[0043] In conclusion, the microcell generation unit (102) is the original image (LF B Dividing and rearranging ) to form the first microcells (M B Generate ) and edit images (LF W Dividing and rearranging ) to form the second microcells (M W ...can generate .... That is, the light field data processing system does not process SAI directly but processes data in units of cells divided by space and angle, thereby providing scalability to process high-resolution light field data while saving memory resources.

[0044] Meanwhile, the microcell generation unit (102) uses original images (LF B ) and edited images (LF W As long as microcells having angular consistency are generated while maintaining spatial features from ), the method of generating said microcells is not limited to the above method and can be varied in many ways.

[0045] Continuing, the image composition unit (104) consists of microcells (M B and M W Combining ) to create a light field image (LF R ) can be generated (S206).

[0046] According to one embodiment, the image composition unit (104) may use a Vision Transformer (ViT) as illustrated in FIG. 5. This ViT may include encoders composed of four stages of transformer blocks, projections, and a decoder composed of four stages of fused layers.

[0047] The first microcells (M B ) is input to the first encoder, and the second microcells (M W ) can be input to the second encoder. Specifically, the first microcells (M B ) and second microcells (M W ) can be separated into patches of size (16×16), for example, and transformed into tokens through the operation of patch embedding and transformer blocks. In this case, the size of the patch can be set as the multiply of the cell's angular resolution.

[0048] For example, the microcell of the present invention may have a size of (512×512) [(128×128) spatial resolution and (4×4) angular resolution], and accordingly, the patch may have a size of (16×16), which is a multiplication of (4×4). This means that pixels requiring angular consistency are grouped within a single patch. Therefore, the results of the encoder can have angular consistency.

[0049] As a result of the operation of this encoder, the first microcells (M B The first tokens (t) corresponding to ) B ) is generated, and second microcells (M W The second tokens corresponding to ) (t W ) can be generated.

[0050] The first tokens (t) output from the transformer block of each stage B ) and the second tokens (t W ) is input as the corresponding projection, and the projection can generate a 2D feature map according to the input. Specifically, the projection consists of first tokens (t B Generate a feature map for ) and the second tokens (t W A feature map for ) is generated, and then the above feature maps can be combined to generate a single feature map (f). At this time, the tokens (t) of the transformer block of each step B and t W Each of these is input to the corresponding projection, and the feature map (f) output from each projection can be input to the corresponding fused layer of the decoder.

[0051] The feature map (f) output from the above projection is adjusted to the size of each stage of the decoder through a (1×1) convolution process.

[0052] Each fused layer performs an element-wise sum with the output of the previous fused layer. At this time, each fused layer sends the output after upsampling and convolution to the next fused layer. As a result, the output from the bottommost fused layer to the topmost fused layer can be added together.

[0053] Passing through these four stages of fused layers yields a single micro light field image (M R ) can be generated.

[0054] Since the above process was performed on a single cell, to apply it to all cells, the above micro light field image (M R The process of generating ) (N s ×N a When repeated ) times, the image composition unit (104) has light field images (LF R ) can be generated. That is, the image composition unit (104) can restore a high-resolution light field image.

[0055] Meanwhile, the image composition unit (104) consists of microcells (M B and M W Combining ) to create a light field image (LF R As long as it can generate ), various models can be used, not limited to the above ViT.

[0056]

[0057] Meanwhile, the components of the aforementioned embodiments can be easily identified from a process perspective. That is, each component can be identified as a separate process. Furthermore, the processes of the aforementioned embodiments can be easily identified from the perspective of the device components.

[0058] In addition, the technical details described above may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either individually or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiments, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. Hardware devices may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

[0059] The embodiments of the present invention described above are disclosed for illustrative purposes only, and those skilled in the art with ordinary knowledge of the present invention may make various modifications, changes, and additions within the spirit and scope of the present invention, and such modifications, changes, and additions should be considered to fall within the scope of the following claims.

Claims

1. An editing department that creates edited images by editing original images; A microcell generation unit that divides the original images to generate first microcells and divides the edited images to generate second microcells; and A light field data processing system characterized by including an image component that generates light field images by combining the first microcells and the second microcells.

2. A light field data processing system according to claim 1, wherein the microcell generating unit divides the original images to generate first cells and rearranges the generated first cells to generate first microcells having a size smaller than the first cells, divides the edited images to generate second cells, and rearranges the generated second cells to generate second microcells having a size smaller than the second cells.

3. A light field data processing system according to claim 1, wherein the first microcell or the second microcell has angular consistency while maintaining spatial features.

4. A light field data processing system according to claim 1, wherein the microcell generating unit performs angle division on the original image or the edited image and then performs spatial division to generate cells having a spatial resolution and angle resolution lower than that of the original image or the edited image, and rearranges the cells to generate the microcells.

5. In paragraph 1, the image component comprises an encoder, a projection, and a decoder, wherein The encoder generates first tokens by encoding the first microcells and generates second tokens by encoding the second microcells, and The above projection generates a feature map by combining the above first tokens and the above second tokens, and A light field data processing system characterized by the above decoder passing the above feature map through a fused layer to generate a micro light field image.

6. A microcell generation unit that generates cells by spatially and temporally dividing images acquired from multiple cameras, and generates microcells smaller than the cells by rearranging the generated cells, wherein A light field data processing system characterized by the above-mentioned microcell having angular consistency while maintaining the spatial features of the image.

7. A light field data processing system according to claim 6, wherein the microcell generation unit performs angle division on the image and then performs spatial division to generate cells having a spatial resolution and angle resolution lower than that of the image.

8. A step of generating edited images by editing original images; A step of dividing the above original images to generate first microcells; A step of dividing the above edited images to generate second microcells; and A light field data processing method characterized by including the step of generating light field images by combining the first microcells and the second microcells.

9. In claim 8, the step of generating the first microcells is, A step of generating first cells by dividing the above original images into spaces and angles; and The method includes the step of rearranging the first cells to generate the first microcells, The step of generating the second microcells above is, A step of generating second cells by dividing the above edited images into spaces and angles; and A light field data processing method characterized by including the step of rearranging the second cells to generate the second microcells.

10. In claim 8, the step of generating the light field images is, A step of encoding the first microcells to generate first tokens; A step of encoding the above second microcells to generate second tokens; A step of generating a feature map by combining the first tokens and the second tokens; and A light field data processing method characterized by including the step of generating a micro light field image by passing the above feature map through a fused layer.

11. In paragraph 8, the step of generating the above-mentioned edited images is, A step of obtaining an edited SAI(I) by editing the above original images; A step of inferring a disparity map (D) of an edited SAI(I) using mono depth estimation; and A light field data processing method characterized by including the step of generating the edited images using the inferred disparity map (D).

Images