Methods and apparatus for crosstalk-free integral imaging based light field displays using directional illumination sources

Abstract

Directional illumination devices and associated methods are described that enable control of the light emission direction from micro-display panels devices such as head mounted displays. These directional illumination designs reduce or eliminate crosstalk. One example directional illumination unit produces a plurality of directional light bundles, where each directional light bundle is confined within a specific angle with respect to an optical axis of head mounted display, and each ray bundle only passes through only one microlens a first microlens array where elemental images associated with the micro-display of the head mounted display are formed. The directional illumination unit includes at least a second microlens array and one or more of a third microlens array, a barrier array, a condenser lens, or a lightguide.

Description

PCT Patent ApplicationAttorney Docket No. 044974.8147 ,WOOO\184774956.1(UA25-157)METHODS AND APPARATUS FOR CROSSTALK-FREE INTEGRAL IMAGING BASED LIGHT FIELD DISPLAYS USING DIRECTIONAL ILLUMINATION SOURCESCROSS-REFERENCE TO RELATED APPLICATION(S)

[0001] This application claims priority to the provisional application with serial number 63 / 736,462 titled “Methods and apparatus for crosstalk-free integral imaging based light field displays using directional illumination sources,” filed December 19, 2024. The entire contents of the above noted provisional application are incorporated by reference as part of the disclosure of this document.TECHNICAL FIELD

[0002] The technology in this patent document relates to integral imaging based light field displays and particularly head-mounted displays.BACKGROUND

[0003] Integral imaging based light field head mounted displays provide a promising solution to reconstruct a true three-dimensional (3D) scene with correct accommodation cues for mitigating the well-known vergence-accommodation conflict. One key challenge that still needs to be addressed is the intrinsic crosstalk issue in conventional integral imaging-based (INI-based) light field head-mounted displays (LF- HMDs) due to the omni-direction emission in typical micro-display panels.

[0004] SUMMARY

[0005] The disclosed embodiments, among other features and benefits, describe methods and apparatus for implementing different configurations of directional illumination sources that, inter alia, enable effective control of the light emission direction from micro-display panels. These directional illumination designs enable the development of crosstalk free integral imaging based light field displays.-1-044974.8147. WOOOM 84774956.1PCT Patent ApplicationAttorney Docket No. 044974.8147 ,WOOO\184774956.1(UA25-157)BRIEF DESCRIPTION OF THE DRAWINGS

[0006] FIG. 1 illustrates an example monocular configuration of a typical Inl-based LF-HMD system.

[0007] FIG. 2 illustrates an Inl-based light field head mounted display in accordance with an example embodiment.

[0008] FIG. 3 illustrates effective solid angle of light emission from a micro-display panel with omnidirectional emission.

[0009] FIG. 4 illustrates a configuration of a transmissive-type micro-display with a directional backlight unit based on micro-Kohler illumination in accordance with an example embodiment.

[0010] FIG. 5 illustrates additional details regarding the directional backlight module of FIG. 4 in accordance with an example embodiment.

[0011] FIG. 6 illustrates an example plot of collection efficiency versus f-number of a condenser lens for a Lambertian emission profile.

[0012] FIG. 7 illustrates an Inl-based light field head mounted display in accordance with another example embodiment.

[0013] FIG. 8 illustrates an Inl-based light field head mounted display in accordance with yet another example embodiment.

[0014] FIG. 9 illustrates an Inl-based light field head mounted display in accordance with another example embodiment.

[0015] FIG. 10 illustrates a bench-top prototype implemented based on the configuration of FIG. 4.

[0016] FIG. 11 illustrates an image of an illuminated area on the back focal plane at the back focal plane of the field lens microlens array MLAFLassociated with the system in FIG. 10.

[0017] FIG. 12 illustrates images of a captured 3D scene in a conventional mode without directional backlights.

[0018] FIG. 12 illustrates images of a captured 3D scene in a conventional mode without directional backlights, showing crosstalk in the images.-2-044974.8147. WOOOM 84774956.1PCT Patent ApplicationAttorney Docket No. 044974.8147 ,WOOO\184774956.1(UA25-157)

[0019] FIG. 13 illustrates images of a captured 3D scene using the system of FIG. 10 using directional illumination, showing no discernable crosstalk in the images.

[0020] DETAILED DESCRIPTION

[0021] In recent years, integral imaging (Ini) based light field (LF) displays have been applied to head-mounted displays (HMD) to generate quasi-natural 3D scenes and thus mitigate the well-known vergence-accommodation conflict (VAC) issue in stereoscopic 3D displays. In a conventional Inl-based head mounted light field display (LF-HMD), a micro lens array (MLA) is placed in front of a micro-display panel. An array of elemental images (Els) is rendered on the display panel. By doing so, each microlens with the corresponding elemental image can generate a different perspective view of a 3D scene. These different perspective views integrally reconstruct the 3D scene with full parallax.

[0022] FIG. 1 illustrates the monocular configuration of a typical Inl-based LF-HMD system, which includes a micro-display panel, an MLA and an eyepiece. A binocular LF-HMD system can be created by adopting a pair of monocular configurations, one for each eye. A set of elemental images (Els) containing different perspective views of a 3D scene are displayed on the micro-display. Each lenslet of the MLA corresponds to an El on the micro-display and forms a conjugate image of the El on the central depth plane (CDP) to create one directional sample of the reconstructed 3D scene. Here the CDP refers to the plane optically conjugate to the micro-display plane through the MLA. The reconstructed 3D scene is viewed at the viewing window (also known as the exit pupil of the eyepiece) by an observer through an eyepiece providing the appropriate depth information. A distinct feature of a light field 3D display from a conventional 2D display is that multiple distinct elemental views rendering a 3D scene point (e.g., P) are observed by placing the pupil of the eye at the viewing window and these views integrally form the retinal image perception of the 3D scene. The accommodated status of the observer's eye plays a critical role on the perceived image. For instance, FIG.1 illustrates the rendering of a 3D point O through three different pixels, Oi, O2, and O3, on the corresponding elemental images. Imaged by three corresponding microlenses, Mi, M2 and M3, the ray bundles from the corresponding points (pixels) on different Els converge to the point O and are further projected on the eye pupil through an eyepiece.-3-044974.8147. WOOOM 84774956.1PCT Patent ApplicationAttorney Docket No. 044974.8147 ,WOOO\184774956.1(UA25-157)When the eye is accommodated at the depth of the reconstruction point O of the reconstructed 3D scene, the ray bundles from the corresponding points (pixels) on different Els will converge to a focused image on the retina, O', as illustrated in FIG. 1. For reconstructed points at other depths (e.g., point P), the images of the individual pixels will be spatially displaced from each other on the retina and will create a retinal blur.

[0023] Two types of micro-display panels can be adopted in an LF-HMD system. The first type is an active light emission device such as a light emitting diode (LED) array, organic light emitting diode (OLED) array, a mini- or micro-LED array and the like. The second type is a passive spatial light modulation (SLM) device combined with an illumination unit where the SLM modulates the light produced by the illumination unit via either reflection or transmission and renders light patterns as images. Nevertheless, the images rendered by both types of micro-display panels exhibit quasi-omnidirectional emission, with light spreading over a wide angular range. Consequently, in a conventional Inl-based LF display, the light rays from a given elemental image not only pass through its own corresponding microlens but also other adjacent microlenses. Consequently, light rays passing through the corresponding microlens form a desired image, but crosstalk images are simultaneously formed through these MLA elements adjacent to the designated MLA-EI pair. As illustrated in FIG. 1 , the light rays from the pixel Oi not only form the desired image 0 / through the corresponding lenslet Mi but also a ghost image Oi" through the lenslet Mo adjacent to the M1 . The Oi" is referred to as a crosstalk image and other similar crosstalk images can be formed through other lenslets adjacent to the lenslet Mi. These crosstalk images not only reduce the overall contrast and resolution of the reconstructed 3D scene, but also lead to sources of error for eye accommodation responses.

[0024] In our previous study, we explained that crosstalk images can be reduced by placing a physical aperture array between the micro-display panel and MLA. However, it can potentially cause vignetting and thus produce other image artifacts. Therefore, to intrinsically remove crosstalk images, it is required to make lights from an elemental image to only pass through the corresponding microlens. According to the disclosed embodiments, to implement this technique, a directional backlight is introduced to control the angular range of the light emission from the micro-display panel. Among other features and benefits, by leveraging what is called Kohler-4-044974.8147. WOOOM 84774956.1PCT Patent ApplicationAttorney Docket No. 044974.8147 ,WOOO\184774956.1(UA25-157) illumination for direct-view 3D displays, in this patent document, we describe several methods and apparatus for implementing different configurations of directional backlights that enable the effective control of the light emission direction from microdisplay panels. These directional backlighting designs enable the development of crosstalk free integral imaging based light field displays. Example embodiments have been analyzed and implemented as well as experimentally validated. In addition to the ability to fully eliminate crosstalk images, optically controlling the light directions can also reduce the waste of light and further improve the overall efficiency of power consumption. For Ini based light field displays, every light ray that does not pass through the designated microlens can be considered as waste of light. Therefore, by limiting light from elemental images to only get through the designated microlens with directional backlights, all the light rays from micro-display panel are effectively used to generate the true-3D scene for users.Example Integral Imaging Based Light Field Head Mounted Displays with Directional Backlights

[0025] FIG. 2 shows the schematic of a crosstalk-free Ini based light field head mounted display in accordance with an example embodiment. The configuration in FIG. 2 includes a micro-display panel with a directional illumination unit, a micro lens array (MLA), and an eyepiece optics. In this example configuration, the micro-display panel is an SLM-type display which transmits or reflects light emitted by the illumination unit. Examples of SLM-type displays include but are not limited to transmissive liquid crystal device (LCD), liquid-crystal-on-silicon (LCoS), or digital mirror device (DMD). The directional illumination unit is typically placed behind a transmissive type SLM to create directional backlights as shown in FIG. 2 or is placed in front of a reflective type SLM to create frontlights.

[0026] The micro-display with a directional illumination unit controls the light emission directions of individual pixels. As illustrated in FIG. 2, each pixel of the microdisplay emits light rays exclusively with a confined angle, Q, so that the rays from a given pixel can only pass through its corresponding microlens to form the desired image. The directional illumination unit ensures that no light from a given pixel reaches adjacent microlenses to produce crosstalk images so that the crosstalk-free system can be effectively implemented. The micro-display is to render different sets of elemental-5-044974.8147. WOOOM 84774956.1PCT Patent ApplicationAttorney Docket No. 044974.8147 ,WOOO\184774956.1(UA25-157) images (e.g., Elo, Eh, etc.), each of which provides a perspective view of the 3D scene. The micro-display is placed at a distance g away from the MLA. The MLA includes an array of microlenses with the same focal length. Each of the elemental images rendered on the micro-display is imaged through a corresponding microlens of the MLA onto a central depth plane (CDP). Depending on the transverse magnification of the microlenses, the conjugate images of the elemental images may overlap on the CDP. By changing the perspective contents of each El, objects at different depths can be rendered. The eyepiece, which is placed at a distance zo away from the CDP, magnifies the reconstructed 3D scene formed by the integral imaging unit and images it into the visual space. The eyepiece projects the ray bundles from the reconstructed 3D scene onto a viewing window where an observer places the eye pupil to observe the magnified virtual 3D scene.

[0027] Besides the main advantage of eliminating crosstalk, a micro-display integrated with a directional illumination unit has a much higher light efficiency than a conventional micro-display with quasi-omnidirectional emission. FIG. 3 illustrates a conventional display with quasi-omnidirectional emission with a solid angle of approximately TT. Only a small portion of the rays emitted by a pixel, with an effective solid angle of G, is collected by its corresponding lenslet while most of the rays go to adjacent or neighboring microlenses and cause crosstalk images, which can be taken as wasted power. When a directional illumination unit is applied, all the rays from a pixel within the confined angular range are collected by its corresponding lenslet, leading to much higher light efficiency. The throughput efficiency is defined as the ratio of the effective light throughput of a lenslet to the light throughput of an omni-directional micro-display. As an example, for a light source with quasi-omnidirectional emission with a solid angle of approximately TT illuminating an MLA with an F / # of 3, the throughput efficiency of each lenslet is approximately 3%.Example Embodiment 1 : Micro-Kdhler Illumination as Directional Backlights

[0028] To implement directional backlights in Ini LF-HMDs, the micro-Kdhler illumination is one possible embodiment. FIG. 4 shows the schematic of a transmissivetype micro-display with a directional backlight unit based on micro-Kdhler illumination and its integration with Inl-based head-mounted light field displays. The micro-Kohler illumination is adopted to control the light emission direction of the micro-display panel.-6-044974.8147. WOOOM 84774956.1PCT Patent ApplicationAttorney Docket No. 044974.8147 ,WOOO\184774956.1(UA25-157)The directional backlight unit includes a planer Lambertian area source, a condenser lens, and an MLA as a field lens denoted as MLAFL. The planar area source is positioned at the front focal plane of the condenser lens, causing the light emitted from the source to become collimated. The angular range of the collimated light after the condenser lens can expressed as:where Hsis the size of the planar light source. It should be noted that while in the example configurations described herein planar sources are used, it is possible within the scope of the present embodiments to use a non-planar area source with appropriate additional optics to provide the desired emission profile.

[0029] The transmissive micro-display panel attached to MLAFLis placed after the condenser lens. The angular range of the collimated light by the condenser lens, ^coiumated’ determines the angular range of the light emission from the transmissive micro-display panel. As shown in FIG. 5, MLAFL, as a field lens, bends the ray directions from the condenser lens so that the light rays emitted from each pixel of the microdisplay are directed toward the corresponding lenslet of the MLA for light field rendering, MLAEI, and no light can pass through the adjacent microlenses. After the collimated light passing through MLAFL, it will be re-focused to form an array of images of the area source. The images of the area source have equal size and the size of each image in the array is expressed as:

[0030] To fully eliminate crosstalk, the optical design of the directional illumination unit based on micro-KdhIer illumination principle needs to satisfy several conditions. First, it is critical to ensure that the image of the area source is approximately co-located with MLAEI used for light field rendering, which requires the focal length of MLAFLto satisfy the requirement: f MLAFL=9 (3)Where g is the gap between the micro-display panel and MLAEIThe gap, g, is a critical parameter that can affect system performance of Inl-based LF-HMDs, such as angular resolution, viewing window size, view density and so on. It should be noted, however,-7-044974.8147. WOOOM 84774956.1PCT Patent ApplicationAttorney Docket No. 044974.8147 ,WOOO\184774956.1(UA25-157) that given factors such as the thickness of the MLA, the shape of lenslets, manufacturing and alignment tolerances, and the like, there can be some deviations or tolerances associated with the Equation (3). This is generally true for all optical system implementations.

[0031] Second, MLAFLand MLAEIneed to have the same pitch, PMLA, which is the center-to-center distance between two adjacent MLAs:

[0032] Finally, the angular range of the light rays generated by the illumination unit needs to match with the required angular range of the light rays for the light field rendering, which is determined as:where DMLAis the effective diameter of each microlens in the MLAEI, which is determined by effective numerical aperture (NA) of elemental views. Therefore, according to Equations (1 ) to (5), the size of the light source, Hs, needs to be adjusted to ensure the illuminated area, Hilluminatd, is equal to the effective diameter, DMLA.

[0033] Under these requirements outlined above for the micro-Kdhler illumination design, the micro-display panel with a directional illumination unit can replace the typical micro-display panel in Inl-based LF-HMDs field displays and create a crosstalk free light field system.

[0034] The required angular range of the collimated light is related to the effective solid angle of emission from a pixel in micro-display panel. In this example micro-Kdhler illumination design, all light passing through the condenser lens is utilized for forming the desired images. Consequently, the collection efficiency of the condenser lens determines the overall light efficiency of the display system. As illustrated in FIG. 6, assuming the light source has a Lambertian emission profile, the collection efficiency of the condenser lens increases with a decreasing f-number. However, a lower f-number also introduces more aberrations, potentially degrading the illumination performance.-8-044974.8147. WOOOM 84774956.1PCT Patent Application Attorney Docket No. 044974.8147 ,WOOO\184774956.1 (UA25-157)Example Embodiment 2: Array-Based Directional Illumination System Using a Single Condenser Lenslet Array

[0035] To avoid the use of a bulky condenser lens, FIG. 7 illustrates the schematic of an alternative embodiment for a directional illumination unit. The directional illumination unit comprises a Lambertian area source, MLA1 , and a barrier array, which collectively provide directional backlight to a transmissive micro-display attached to MLA1 . The area (2-D) source can be a continuous planar source with its size equal to or greater than the size of MLA to cover the entire area of the micro-display. The area source can also be made of an array of small area sources, one for each lenslet of MLA1 . An array of barriers, one for each lenslet, is inserted between the area source and MLA1 to ensure rays for a given source region can only pass through its corresponding lenslet in MLA1 , blocking rays from reaching adjacent lenslet. The transmissive micro-display is directly attached to MLA1 to receive the illumination from the source. The micro-display may be placed in front of or behind MLA1 .

[0036] In this example configuration, MLA1 serves as a condenser lenslet array through which the light source is imaged into an array of area sources. T o fully eliminate crosstalk, it is critical to ensure that the image of the area source via MLA1 is approximately co-located with MLAEI used for light field rendering, which requires the focal length of MLA1 satisfying the requirement:Where g is the gap between MLA1 and MLAEI, and Zs is the signed distance of the source from the MLA1 , and fMLAi is the focal length of MLA1 . Equation (6) ensures that the area source is optically conjugated to the location of the microlens array used for light field rendering (MLAEI). Additionally, the aperture size of the barrier array, Hs, needs to satisfy the requirement:Where Hmuminated is the size of the effective aperture area on the MLAEI to be illuminated. The condition expressed in Equation (7) ensures that a specific area on the area source confined by the barrier array aperture is imaged onto the effective aperture of the MLAEI. The range of rays passing through each pixel is thus restricted by the corresponding barrier element.-9-044974.8147. WOOOM 84774956.1PCT Patent ApplicationAttorney Docket No. 044974.8147 ,WOOO\184774956.1(UA25-157)

[0037] Under these requirements expressed in Equations (6) and (7), the focal length of MLA1 is generally shorter than the gap between the two MLAs and in a preferred embodiment it is equal to g / 2 to maximize the use of the aperture of the M I_AEI. Additionally, the size of each elemental image is slightly smaller than the pitch of the microlens array due to the presence of the barrier (e.g., to accommodate, for example the thickness of the barrier vanes).Example Embodiment s: Array-Based Directional Illumination System Using a Double- MLA Pair

[0038] One of the main challenges of the design shown in FIG. 7 is that the design requires a short focal length for the MLA1. To overcome this challenge, FIG. 8 shows the schematic layout of an alternative design modified from the design in FIG. 7. The directional illumination unit includes an array of Lambertian area source, a pair of MLAs (MLA1 and MLA2), and a barrier array, which collectively provide directional backlighting to a transmissive micro-display attached to MLA1. The area source is made of an array of small area sources, one for each lenslet of the MLA pair. The area source can also be a continuous planar source with its size equal to or greater than the size of the MLAs to cover the entire area of the micro-display. If a large, continuous area source is used, light rays from small segments of source, marked in black, would be blocked by the barrier array and thus wasted.

[0039] MLA2, with a focal length of fMLA2, is inserted between the source and the MLA1 . In a preferred embodiment, the area source is placed within the front focal point of the MLA2, satisfying the following:\^s \ — f MLA (8)Where Zs is the spacing between the area source and MLA2.

[0040] To fully eliminate crosstalk, the images of the area source via the combination of MLA1 and MLA2 are approximately co-located with MLAEI used for light field rendering, and it is required that a specific area on the area source, Hs, is imaged onto the effective aperture of the M LAEI. A barrier array is optionally inserted between MLA1 and MLA2 to minimize stray light. While the example configuration in FIG. 8 illustrates that the distance, g, between the micro-display and MLAEI is larger than / MLAI , unlike Embodiment 2 shown in FIG. 7, the focal lengths of MLA1 and MLA2 can be-10-044974.8147. WOOOM 84774956.1PCT Patent ApplicationAttorney Docket No. 044974.8147 ,WOOO\184774956.1(UA25-157) significantly longer than g, which provides additional design flexibility. Though not required, the focal lengths of MLA1 and MLA2 can be equal as well. The spacing between the two microlens arrays (t) is determined by the effective numerical aperture (NA) of the ray bundle in the MLA2 image space.Example Embodiment 4: Directional Illumination System Using a Substrate-Guided Waveguide or Lightguide

[0041] To avoid the use of a bulky condenser lens, FIG. 9 illustrates the schematic of yet another alternative embodiment for a directional illumination unit. The directional illumination unit comprises an area source, a condenser lens, a substrate-guided waveguide or lightguide, and a MLA1 , which collectively provide directional backlights to a transmissive micro-display attached to the MLA1. An enlarged view of the lower section of the directional illumination unit is shown on the left side of the figure. The area source in this example is a continuous planar source. The planar area source is positioned at the front focal plane of the condenser lens, causing the light emitted from the source to become collimated. The angular range of the collimated light after the condenser lens can expressed as:where Hsis the size of the planar light source. Different from the embodiment in FIG. 4, here the size of the area source and the focal length of the condenser lens are much smaller, as the diameter of the collimated beam only needs to be large enough to match the effective aperture of a single lenslet of MLA1 , ratherthan the entire area of the MLA1 or the micro-display.

[0042] The substrate-guided waveguide or lightguide includes a substrate, an incoupler, and an out-coupler. The collimated light rays from the condenser are coupled into the substrate via the in-coupler which directs the in-coupled rays toward the substrate for further propagation. The in-coupled rays are guided within the substrate via total internal reflection until portions of the rays are out-coupled toward the lenslet at each intersection with the out-coupler.

[0043] The transmissive micro-display is directly attached to MLA1 to receive the illumination from the source. The micro-display may be placed in front of or behind MLA1. The angular range of the collimated light by the condenser lens, <Coiiimated ,-11-044974.8147. WOOOM 84774956.1PCT Patent ApplicationAttorney Docket No. 044974.8147 ,WOOO\184774956.1(UA25-157) determines the angular range of light emission from the transmissive micro-display panel. MLA1 as a field lens bends the ray directions from the condenser lens so that the light rays emitted from each pixel of the micro-display are directed toward the corresponding lenslet of the M LA for light field rendering, MLAEI, and no light can pass through the adjacent microlenses. After the collimated light passing through MLA1, it will be re-focused to form an array of images of the area source. The images of the area source have equal size and the size of each image in the array is expressed as:

[0044] To fully eliminate crosstalk, the optical design of the directional illumination unit needs to satisfy several conditions. First, it is critical to ensure that the image of the area source is approximately co-located with MLAEI used for light field rendering, which requires the focal length of MLA1 to satisfy the requirement: f MLA1=9 (11)Where g is the gap between the micro-display panel and MLAEI.

[0045] Second, MLA1 and MLAEIneed to have the same pitch, PMLA, which is the center-to-center distance between two adjacent MLAs:PMLAFL — MLAEI — PMLA (12)

[0046] The waveguide-based illumination system offers a more compact alternative to embodiment 1. In this configuration, a lightguide or waveguide is introduced as a pupil expander, replacing the large condenser used in embodiment 1. The condenser collimates light from the small Lambertian source, and the collimated light is then coupled into the waveguide. Within the waveguide, the collimated light bundle is duplicated, allowing the coverage of collimated light to become sufficiently large for the entire micro-display. This expanded coverage of collimated light exits the waveguide and passes through the MLA1 and micro-display, ultimately illuminating the MLA-EI. However, this approach presents some challenges. Issues such as ghost images and light loss during pupil duplications can arise, potentially degrading the overall illumination performance.-12-044974.8147. WOOOM 84774956.1PCT Patent ApplicationAttorney Docket No. 044974.8147 ,WOOO\184774956.1(UA25-157)Example Experimental Results

[0047] Based on the configuration in FIG. 4, we implemented a proof-of-concept prototype for crosstalk-free Inl-based LF-HMD system as shown in FIG. 10. To build the micro-display with directional backlights, a microOLED panel was used as the planar light source. Then, a condenser lens with 36mm focal length, a transparent LCD panel, and an MLA with 4mm focal length as MLAFieldwere placed within the system according to the configuration of FIG. 4. Following the micro-display with directional backlights, another MLA with 3.3mm focal length as MLAEIwas positioned, followed by an eyepiece and a camera.

[0048] To validate the performance of micro-Kbhler illumination, the image at the back focal plane of MLAFLwere captured, as shown in Figure 11 . During the validation, MLAEIand the transparent LCD are removed so that the eyepiece can magnify the image at the back focal plane of MLAFL. The square illuminated area is replicated for each microlens on MLAEI. Therefore, when MLAEIis placed at this back focal plane, each microlens on MLAEIcan image its own elemental image to form a true-3D scene without any crosstalk image.

[0049] The crosstalk-free concept is demonstrated by capturing light field images in both conventional mode and crosstalk-free mode. The main optical specifications of the display system are shown in Table 1 . In the conventional mode, the condenser lens is removed, and the OLED micro-display is attached with the transparent LCD as its backlight so that the angular emission range spread across different directions. FIG. 12 presents the captured 3D scene in conventional mode without directional backlights. The 3D scene includes of three sets of resolution targets placed at three different depths: 2.5 diopters, 1 diopter, and 0.1 diopters, from left to right, respectively. Even though the 3D scene is well-reconstructed, crosstalk images are noticeably present in the peripheral regions as highlighted in the zoomed-in box in FIG. 12. In FIG. 13, the image of the same reconstructed 3D scene captured in crosstalk-free mode with directional backlights. Compared to FIG. 12, there are no crosstalk images in FIG. 13.-13-044974.8147. WOOOM 84774956.1PCT Patent ApplicationAttorney Docket No. 044974.8147 ,WOOO\184774956.1(UA25-157)Table 1. Specifications of the System.

[0050] One aspect of the disclosed embodiments relates to a method for improving operations of an integral imaging based light field head mounted display comprising using a directional backlight to provide effective control of light emission direction from a micro-display panel to produce integral imaging based light field displays with reduced or no crosstalk. Example embodiments also relate to reducing waste of light and improving an overall efficiency of power consumption achieved at least in-part by limiting light from elemental images to only get through a designated microlens with directional backlights, thereby using all or a substantial portion of light rays from the micro-display panel to generate a true-3D scene.

[0051] One aspect of the disclosed embodiments relates to an integral imagingbased light field head mounted display (Inl-based LF-HMD) that includes a microdisplay, and a directional illumination unit that includes one or more area light sources, where the directional illumination unit is positioned to provide illumination to the microdisplay. The Inl-based LF-HMD further includes a first microlens array positioned to receive light from the micro-display, each lenslet of the first microlens array corresponding to an elemental image on the micro-display and positioned to form a-14-044974.8147.WOOOM 84774956. 1PCT Patent ApplicationAttorney Docket No. 044974.8147 ,WOOO\184774956.1(UA25-157) conjugate image of the corresponding elemental image on a central depth plane. The Inl-based LF-HMD further includes an eyepiece positioned to receive light from the first microlens array after passing through the central depth plane to enable generation of a three-dimensional (3D) scene. The directional illumination unit is configured to produce a plurality of directional light bundles, each directional light bundle confined within an angle, 6, with respect to an optical axis of the Inl-based LF-HMD and only passing through one corresponding microlens of the first microlens array.

[0052] In one example embodiment, the directional illumination unit includes at least a second microlens array and one or more of: a third microlens array, a barrier array, a condenser lens, or a lightguide.

[0053] In another example embodiment, the directional illumination unit includes a second microlens array positioned at one side of the micro-display and a condenser lens positioned between the one or more area light sources and the condenser lens. In this example embodiment there is a one-to-one correspondence between each microlens of the first microlens array and each microlens of the second microlens array, the one or more area light sources are positioned at a focal plane of the condenser lens to enable the condenser lens produce a collimated light in a direction of the second microlens array, the condenser lens is dimensioned to allow the collimated light to illuminate a full extent of the second microlens array, and the second microlens array is configured as a field lens to bend ray directions of the collimated light received on the lenslets of the second microlens array toward corresponding lenslets of the first microlens array. According to one example embodiment, a distance between the first microlens array and the second microlens array is equal to a focal length associated with the second microlens array, and the first and the second microlens arrays have a same pitch. In yet another example embodiment, the one or more area light sources consist of a planer light source, an angular range of the collimated light is a function of a dimension of the planar light source and a focal length of the condenser lens, and the second microlens array is positioned to produce an array of images of the planar light source at each lenslet of the second microlens array. In still another example embodiment, the dimension of the planar light source is selected to allow an effective diameter of each microlens of the first microlens array to be fully illuminated by a corresponding lenslet of the second microlens array without illuminating any other lenslet of the first microlens array. In one example embodiment,-15-044974.8147. WOOOM 84774956.1PCT Patent ApplicationAttorney Docket No. 044974.8147 ,WOOO\184774956.1(UA25-157)

[0054] According to another example embodiment, the directional illumination unit includes a second microlens array positioned at one side of the micro-display and a barrier array positioned between the one or more area light sources and the second microlens array. In this example embodiment, there is a one-to-one correspondence between each microlens of the first microlens array and each microlens of the second microlens array, and the barrier array includes multiple subsections, each subsection extending from one microlens of the second microlens array in a direction of the one of more area light sources to confine illumination from a section, or from one of, the one or more area light sources to only one microlens of the second microlens array. In one example embodiment, the one or more area light sources consist of one planar light source, and each subsection of the barrier array extends to one section of the planer light source. In another example embodiment, the one or more area light sources consist of a plurality of planar light sources, and each subsection of the barrier array extends to one of the plurality of the planer light sources. In still another example embodiment, the second microlens array is configured as a condenser lens array, and the second microlens array is positioned to produce images of the one or more area light sources that are co-located with the first microlens array. In one example embodiment, a focal length of the sconed microlens array is selected to be equal to where g is a distance between the first and the second microlens arrays, and Zszs~g is a signed distance between of the one or more area light sources and the second microlens array. In another example embodiment, an aperture size of each subsection of the barrier array is less than or equal toHllluminated'^z^ , where Zs is a signed distance9 between of the one or more area light sources and the second microlens array, g is a distance between the first and the second microlens arrays, and Hmuminated is a size of an effective aperture area on the first microlens array to be illuminated. In another example embodiment, a size of each elemental image is smaller than a pitch of the first microlens array.

[0055] In another example embodiment, the directional illumination unit includes a second microlens array that is positioned at one side of the micro-display, a third microlens array and a barrier array, wherein the third microlens array is positioned between the second microlens array and the one or more area light sources, and the barrier array is positioned between the one or more area light sources and the third-16-044974.8147. WOOOM 84774956.1PCT Patent ApplicationAttorney Docket No. 044974.8147 ,WOOO\184774956.1(UA25-157) microlens array. In this embodiment, there is a one-to-one correspondence between each microlens of the first microlens array and each microlens of the second microlens array and each microlens of the third microlens array, the barrier array includes multiple subsections, each subsection extending from one microlens of the third microlens array to a corresponding microlens of the second microlens array to confine illumination provided by the one microlens of the third microlens array for reception by only the corresponding microlens of the second microlens array. Further, the third microlens array is positioned to receive illumination from only one section or from only one of the one or more area light sources. In one example embodiment, the one or more area light sources consist of a plurality of planar area light sources, wherein each of the plurality of planar area light sources is configured to provide illumination for one of the lenslets of the third microlens array. In another example embodiment, the one or more area light sources consist of one planar area light source comprising a plurality of subsections, wherein each subsection of the planar area light source is configured to provide illumination for one of the lenslets of the third microlens array. In yet another example embodiment, a distance between the one or more area light sources and the third microlens array is less than or equal to a focal length of the second microlens array. In still another example embodiment, images of the one or more area light sources produced by a combination of the second and the third microlens arrays are co-located with the first microlens array. In one example embodiment, either or both focal lengths of the second microlens array or the third microlens array are larger than a distance between the first and the second microlens arrays. In another example embodiment, a focal length of the second microlens array and a focal length of the third microlens array are different from one another.

[0056] In another example embodiment, the directional illumination unit includes a second microlens array positioned at one side of the micro-display, a substrate guided waveguide or a light guide, and a condenser lens. In this example embodiment, the one or more area light sources consist of one planar area light source having a dimension that is smaller than an extent of the second microlens array, the condenser lens is positioned to receive light from the planar area light source and to provide collimated light to the substrate guided waveguide or the light guide, the substrate guided waveguide or the light guide allowing light that is received from the condenser lens to propagate therein and to provide light in a direction of the second microlens-17-044974.8147. WOOOM 84774956.1PCT Patent ApplicationAttorney Docket No. 044974.8147 ,WOOO\184774956.1(UA25-157) array, and the second microlens array is configured as a field lens to bend ray directions of light received on lenslets of the second microlens array toward corresponding lenslets of the first microlens array. In one example embodiment, an angular range of the collimated light produced by the condenser lens is a function of a dimension of the planar area light source and a focal length of the condenser lens. In another example embodiment, images of the planar area light source produced at each lenslet of the first microlens array have equal sizes, and a size of each of the images of the planar area light source is directly proportional to a product of a focal length of the second microlens array and a dimension of the planar area light source, and is inversely proportional to a focal length of the condenser lens. In yet another example embodiment, a focal length of the second microlens array is equal to a distance between the first microlens array and the second microlens array. In still another example embodiment, the first microlens array and the second microlens array have a same pitch.

[0057] While this patent document contains many specifics, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this patent document in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0058] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. Moreover, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.-18-044974.8147. WOOOM 84774956.1PCT Patent ApplicationAttorney Docket No. 044974.8147 ,WOOO\184774956.1(UA25-157)

[0059] It is understood that the various disclosed embodiments may be implemented individually, or collectively, in devices comprised of various optical components, electronics hardware and / or software modules and components. These devices, for example, may comprise a processor, a memory unit, an interface that are communicatively connected to each other. The processor and / or controller can perform various disclosed operations based on execution of program code that is stored on a storage medium. The processor and / or controller can, for example, be in communication with at least one memory and with at least one communication unit that enables the exchange of data and information, directly or indirectly, through the communication link with other entities, devices and networks. The communication unit may provide wired and / or wireless communication capabilities in accordance with one or more communication protocols, and therefore it may comprise the proper transmitter / receiver antennas, circuitry and ports, as well as the encoding / decoding capabilities that may be necessary for proper transmission and / or reception of data and other information. For example, the processor may be configured to receive electrical signals or information from the disclosed sensors (e.g., CMOS sensors), and to process information to produce images or other information of interest.

[0060] Various information and data processing operations described herein may be implemented in one embodiment by a computer program product, embodied in a computer-readable medium, including computer-executable instructions, such as program code, executed by computers in networked environments. A computer- readable medium may include removable and non-removable storage devices including, but not limited to, Read Only Memory (ROM), Random Access Memory (RAM), compact discs (CDs), digital versatile discs (DVD), etc. Therefore, the computer-readable media that is described in the present application comprises non- transitory storage media. The instructions may be stored on memory of a local processing device, or may be stored in a remote location, such as a remote server, a cloud sever, or other networked devices and environments. Generally, program modules may include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Computerexecutable instructions, associated data structures, and program modules represent examples of program code for executing steps of the methods disclosed herein. The particular sequence of such executable instructions or associated data structures-19-044974.8147. WOOOM 84774956.1PCT Patent ApplicationAttorney Docket No. 044974.8147 ,WOOO\184774956.1 (UA25-157) represents examples of corresponding acts for implementing the functions described in such steps or processes.

[0061] Only a few implementations and examples are described and other implementations, enhancements and variations can be made based on what is described and illustrated in this patent document.-20-044974.8147. WOOOM 84774956.1

Claims

PCT Patent ApplicationAttorney Docket No. 044974.8147 ,WOOO\184774956.1(UA25-157)CLAIMS l / We claim:1 . An integral imaging-based light field head mounted display (Inl-based LF- HMD), comprising: a micro-display; a directional illumination unit including one or more area light sources, the directional illumination unit positioned to provide illumination to the micro-display; a first microlens array positioned to receive light from the micro-display, each lenslet of the first microlens array corresponding to an elemental image on the microdisplay and positioned to form a conjugate image of the corresponding elemental image on a central depth plane; and an eyepiece positioned to receive light from the first microlens array after passing through the central depth plane to enable generation of a three-dimensional (3D) scene, wherein: the directional illumination unit is configured to produce a plurality of directional light bundles, each directional light bundle confined within an angle, 0, with respect to an optical axis of the Inl-based LF-HMD and only passing through one corresponding microlens of the first microlens array.

2. The Inl-based LF-HMD of claim 1 , wherein the directional illumination unit includes at least a second microlens array and one or more of: a third microlens array, a barrier array, a condenser lens, or a lightguide.

3. The Inl-based LF-HMD of claim 1 , wherein the directional illumination unit includes a second microlens array positioned at one side of the micro-display and a condenser lens positioned between the one or more area light sources and the condenser lens, wherein: there is a one-to-one correspondence between each microlens of the first microlens array and each microlens of the second microlens array,-21-044974.8147. WOOOM 84774956.1PCT Patent ApplicationAttorney Docket No. 044974.8147 ,WOOO\184774956.1(UA25-157) the one or more area light sources are positioned at a focal plane of the condenser lens to enable the condenser lens produce a collimated light in a direction of the second microlens array, the condenser lens is dimensioned to allow the collimated light to illuminate a full extent of the second microlens array, and the second microlens array is configured as a field lens to bend ray directions of the collimated light received on the lenslets of the second microlens array toward corresponding lenslets of the first microlens array.

4. The Inl-based LF-HMD of claim 3, wherein a distance between the first microlens array and the second microlens array is equal to a focal length associated with the second microlens array, and the first and the second microlens arrays have a same pitch.

5. The Inl-based LF-HMD of claim 3 or 4, wherein: the one or more area light sources consist of a planer light source, an angular range of the collimated light is a function of a dimension of the planar light source and a focal length of the condenser lens, and the second microlens array is positioned to produce an array of images of the planar light source at each lenslet of the second microlens array.

6. The Inl-based LF-HMD of claim 5, wherein the dimension of the planar light source is selected to allow an effective diameter of each microlens of the first microlens array to be fully illuminated by a corresponding lenslet of the second microlens array without illuminating any other lenslet of the first microlens array.

7. The Inl-based LF-HMD of claim 1 , wherein the directional illumination unit includes a second microlens array positioned at one side of the micro-display and a barrier array positioned between the one or more area light sources and the second microlens array, wherein: there is a one-to-one correspondence between each microlens of the first microlens array and each microlens of the second microlens array, and-22-044974.8147. WOOOM 84774956.1PCT Patent ApplicationAttorney Docket No. 044974.8147 ,WOOO\184774956.1(UA25-157) the barrier array includes multiple subsections, each subsection extending from one microlens of the second microlens array in a direction of the one of more area light sources to confine illumination from a section, or from one of, the one or more area light sources to only one microlens of the second microlens array.

8. The Inl-based LF-HMD of claim 7, wherein the one or more area light sources consist of one planar light source, and wherein each subsection of the barrier array extends to one section of the planer light source.

9. The Inl-based LF-HMD of claim 7, wherein the one or more area light sources consist of a plurality of planar light sources, and wherein each subsection of the barrier array extends to one of the plurality of the planer light sources.

10. The Inl-based LF-HMD of any of claims 7 to 9, wherein the second microlens array is configured as a condenser lens array, and wherein the second microlens array is positioned to produce images of the one or more area light sources that are co-located with the first microlens array.11 . The Inl-based LF-HMD of claim 10, wherein a focal length of the sconed microlens array is selected to be equal to: g - Zszs- g where g is a distance between the first and the second microlens arrays, and Zs is a signed distance between of the one or more area light sources and the second microlens array.

12. The Inl-based LF-HMD of claim 10, wherein an aperture size of each subsection of the barrier array is less than or equal to:where Zs is a signed distance between of the one or more area light sources and the second microlens array, g is a distance between the first and the second microlens arrays, and Hmuminated is a size of an effective aperture area on the first microlens array to be illuminated.-23-044974.8147. WOOOM 84774956.

113. The Inl-based LF-HMD of any of claims 7 to 9, wherein a size of each elemental image is smaller than a pitch of the first microlens array.

14. The Inl-based LF-HMD of claim 1 , wherein the directional illumination unit includes a second microlens array that is positioned at one side of the micro-display, a third microlens array and a barrier array, wherein the third microlens array is positioned between the second microlens array and the one or more area light sources, and the barrier array is positioned between the one or more area light sources and the third microlens array, wherein: there is a one-to-one correspondence between each microlens of the first microlens array and each microlens of the second microlens array and each microlens of the third microlens array, the barrier array includes multiple subsections, each subsection extending from one microlens of the third microlens array to a corresponding microlens of the second microlens array to confine illumination provided by the one microlens of the third microlens array for reception by only the corresponding microlens of the second microlens array, and the third microlens array is positioned to receive illumination from only one section or from only one of the one or more area light sources.

15. The Inl-based LF-HMD of claim 14, wherein the one or more area light sources consist of a plurality of planar area light sources, wherein each of the plurality of planar area light sources is configured to provide illumination for one of the lenslets of the third microlens array.

16. The Inl-based LF-HMD of claim 14, wherein the one or more area light sources consist of one planar area light source comprising a plurality of subsections, wherein each subsection of the planar area light source is configured to provide illumination for one of the lenslets of the third microlens array.-24-044974.8147. WOOOM 84774956.1PCT Patent ApplicationAttorney Docket No. 044974.8147 ,WOOO\184774956.1(UA25-157)17. The Inl-based LF-HMD of any of claims 14 to 16, wherein a distance between the one or more area light sources and the third microlens array is less than or equal to a focal length of the second microlens array.

18. The Inl-based LF-HMD of any of claims 14 to 16, wherein images of the one or more area light sources produced by a combination of the second and the third microlens arrays are co-located with the first microlens array.

19. The Inl-based LF-HMD of any of claims 14 to 16, wherein either or both focal lengths of the second microlens array or the third microlens array are larger than a distance between the first and the second microlens arrays.

20. The Inl-based LF-HMD of any of claims 14 to 16, wherein a focal length of the second microlens array and a focal length of the third microlens array are different from one another.21 . The Inl-based LF-HMD of claim 1 , wherein the directional illumination unit includes a second microlens array positioned at one side of the micro-display, a substrate guided waveguide or a light guide, and a condenser lens, wherein: the one or more area light sources consist of one planar area light source having a dimension that is smaller than an extent of the second microlens array, the condenser lens is positioned to receive light from the planar area light source and to provide collimated light to the substrate guided waveguide or the light guide, the substrate guided waveguide or the light guide allowing light that is received from the condenser lens to propagate therein and to provide light in a direction of the second microlens array, and the second microlens array is configured as a field lens to bend ray directions of light received on lenslets of the second microlens array toward corresponding lenslets of the first microlens array.

22. The Inl-based LF-HMD of claim 21 , wherein an angular range of the collimated light produced by the condenser lens is a function of a dimension of the planar area light source and a focal length of the condenser lens.-25-044974.8147. WOOOM 84774956.1PCT Patent ApplicationAttorney Docket No. 044974.8147 ,WOOO\184774956.1(UA25-157)23. The Inl-based LF-HMD of claims 21 or 22, wherein images of the planar area light source produced at each lenslet of the first microlens array have equal sizes, and wherein a size of each of the images of the planar area light source is directly proportional to a product of a focal length of the second microlens array and a dimension of the planar area light source, and is inversely proportional to a focal length of the condenser lens.

24. The Inl-based LF-HMD of claims 21 or 22, wherein a focal length of the second microlens array is equal to a distance between the first microlens array and the second microlens array.

25. The Inl-based LF-HMD of claims 21 or 22, wherein the first microlens array and the second microlens array have a same pitch.-26-044974.8147. WOOOM 84774956.1

Images