Dynamic configuration and control of an augmented reality device for reduced power consumption

Abstract

Methods and apparatuses disclosed herein provide for improved operation of an augmented reality (AR) device (10), such as a wearable AR headset or other portable AR device, based on dynamic, intelligent selection of a reduced power mode characterized by reductions in power consumption with respect both to gaze tracking and image generation. These combined power savings yield meaningful improvements in battery life and, advantageously, the device configuration(s) used for presentation of AR images (24) during the reduced power mode complement a reduced gaze tracking resolution that is used during operation of the AR device (10) in its reduced power mode.

Description

[0001] P112065W001

[0002] DYNAMIC CONFIGURATION AND CONTROL OF AN AUGMENTED REALITY DEVICE FOR REDUCED POWER CONSUMPTION

[0003] TECHNICAL FIELD

[0004] Disclosed methods and apparatuses relate to augmented reality (AR) and the dynamic configuration and control of an AR device for reduced power consumption.

[0005] BACKGROUND

[0006] Modern augmented reality (AR) headsets and other AR devices represent a significant advancement in human-computer interaction, combining sophisticated display technology with precise eye tracking systems to overlay digital content onto the user's view of the real world. Such devices typically consist of a head-mounted display unit containing transparent or semi-transparent optical elements, processing hardware, and various sensors for tracking both the user's environment and their eye movements.

[0007] The fundamental operation of AR devices relies on several key technologies working in concert. The optical system typically employs either waveguide displays or transparent OLED panels positioned in front of the eyes of the wearer (user). These display elements must be carefully calibrated to maintain proper alignment with the field of view of the user while allowing natural light to pass through, ensuring the user can still see their physical surroundings. Digital content— the AR image— projected into these optical elements creates the illusion that virtual objects exist within the real environment surrounding the user.

[0008] A critical component of modern AR systems is eye tracking technology, which enables the device to understand exactly where the user is looking at any given moment. Two primary methods are commonly employed for this purpose: video-oculography (VOG) and electrooculography (EOG).

[0009] VOG utilizes small infrared cameras mounted within the headset to capture detailed images of the eyes of the user. These cameras track specific features such as the pupil position, corneal reflection, and iris patterns. By analyzing these features in realtime, the system can determine the gaze direction of the user with high precision. VOG systems typically operate by illuminating the eye with infrared light and measuring the relative positions of the pupil center and corneal reflection, a technique known as pupil center corneal reflection (PCCR) tracking.

[0010] Electrooculography, in contrast, measures the natural electrical potential differences that exist between the front and back of the eye. This potential difference creates an electrical field that changes as the eye rotates within its socket. By placingP112065W001

[0011] electrodes around the eye, EOG systems detect these changes in electrical potential and use them to determine eye movement and position. While EOG typically offers lower spatial resolution compared to VOG, it has the advantages of working well in varying lighting conditions and requiring less computational power.

[0012] The integration of these eye-tracking technologies into AR headsets allows for natural interaction by the user with virtual objects through gaze-based selection and control. Further, eye-tracking allows the AR system to maintain proper alignment and scaling of augmented content relative to the viewpoint of the user, ensuring a convincing and comfortable mixed reality experience.

[0013] Advanced AR systems may combine both VOG and EOG methods to leverage their complementary strengths. However, challenges remain. For example, even where an AR system incorporates both VOG and EOG methods, significant complexities arise in the context of realizing the full potential of power savings associated with EOG-based tracking. It maybe, for example, that the various subsystems of the AR system are designed or otherwise configured to support the high-resolution operation associated with VOG-based tracking, and do not provide meaningful reconfigurability for lower power operation when only EOG-based tracking is active.

[0014] SUMMARY

[0015] Methods and apparatuses disclosed herein provide for improved operation of augmented reality (AR) devices, such as wearable AR headsets or other portable AR devices, based on dynamic, intelligent selection of a reduced power mode characterized by reductions in power consumption with respect to both gaze tracking and image generation. These combined power savings yield meaningful improvements in battery and, advantageously, the device configuration(s) used for presentation of AR images during the reduced power mode complement a reduced gaze tracking resolution that is used during operation of the AR device in its reduced power mode.

[0016] One embodiment comprises an AR device that includes a transparent viewing element positioned in front of one or both eyes of a wearer of the AR device, with the transparent viewing element including an AR image display area. Further included is a gaze tracking system selectively operable in either a first gaze tracking mode having higher resolution or a second gaze tracking mode having lower power consumption, and an electronic display selectively operable in either a full display mode or a partial display mode. The full display mode makes all image pixels available for generating an AR image, and the partial display mode makes only a subset of the image pixels available for generating the AR image and powers down at least a portion of display circuitry inP112065W001

[0017] the electronic display that is associated with the image pixels that are unavailable in the partial display mode.

[0018] Further components of the AR device include a light coupling mechanism configured to couple the AR image to the AR image display area of the transparent viewing element, and device control circuitry. The device control circuitry is configured to determine a currently required gaze tracking resolution for tracking eye movements of the wearer of the AR device, and, responsive to the currently required gaze tracking resolution being satisfied by the second gaze tracking mode, selecting a reduced power mode of the AR device. The device control circuitry selects the reduced power mode by selecting the second gaze tracking mode and selecting the partial display mode.

[0019] A related embodiment comprises a method of operation by an AR device. The method includes the device determining a currently required gaze tracking resolution for tracking eye movements of the wearer of the AR device, where the AR device supports a first gaze tracking mode having a first gaze tracking resolution and a second gaze tracking mode that requires less power than the first gaze tracking mode and has a second gaze tracking resolution that is lower than the first resolution. The method further includes, in response to the currently required gaze tracking resolution being satisfied by the second gaze tracking mode, selecting a reduced power mode of the AR device. Selecting the reduced power mode includes selecting the second gaze tracking mode and selecting a partial display mode of an electronic display used to generate an AR image displayed by the AR device. The electronic display is selectively operable in a full display mode that makes all image pixels available for generating the AR image or in a partial display mode that makes only a subset of the image pixels available for generating the AR image and powers down one or more portions of the electronic display associated with the image pixels that are unavailable.

[0020] Of course, the present invention is not limited to the above features and advantages. Indeed, those skilled in the art will recognize additional features and advantages upon reading the following detailed description, and upon viewing the accompanying drawings.

[0021] BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a block diagram of an augmented reality (AR) device according to one embodiment.

[0023] Figure 2 is a diagram of an example arrangement of graphical elements contained in an AR image, for use with a first gaze tracking mode of an AR device.P112065W001

[0024] Figure 3 is a diagram of an example arrangement of graphical elements contained in an AR image, for use with a first gaze tracking mode of an AR device.

[0025] Figures 4-6 are diagrams of an example array or grid of image pixels of an electronic display that is used selectively either in a full display mode or a partial display mode, according to one embodiment.

[0026] Figures 7 and 8 are block diagrams of an optical path of a light coupling mechanism used in an AR device, according to one embodiment.

[0027] Figure 9 is a block diagram of an optical path of a light coupling mechanism used in an AR device, according to another embodiment.

[0028] Figure 10 is a logic flow diagram of a method of operation by an AR device, according to one embodiment.

[0029] Figure 11 is a block diagram of an electronic display for use in an AR device, according to one embodiment.

[0030] Figures 12 and 13 are block diagrams of further optical path examples, according to another embodiment.

[0031] DETAILED DESCRIPTION

[0032] Figure 1 illustrates example details for an augmented reality (AR) device 10, according to one or more embodiments. In at least one embodiment, the AR device 10 is a wearable headset and it should be appreciated that the AR device 10 may include components beyond those illustrated, such as a battery pack or other power source configured to provide operational power to the various components of the AR device 10, which are also referred to as subsystems or subassemblies.

[0033] Illustrated components of interest include a transparent viewing element 12 positioned in front of one or both eyes of a wearer of the AR device 10. The transparent viewing element 12 includes an AR image display area 14 and in at least one embodiment the transparent viewing element includes left and right optical lenses with respective AR image display areas 14.

[0034] Further components include a gaze tracking system 16 that is selectively operable in either a first gaze tracking mode having higher resolution or a second gaze tracking mode having lower power consumption. For example, a first gaze tracking subsystem 18 provides for operation in the first gaze tracking mode, while a second gaze tracking subsystem 20 provides for operation in the second gaze tracking mode.

[0035] With the first gaze tracking subsystem 18 providing higher resolution than the second gaze tracking subsystem 20 but requiring comparatively higher power, the arrangement provides the option for selecting higher gaze tracking resolution whenP112065W001

[0036] needed, at the expense of increased power consumption. When the higher resolution is not needed, selecting the second gaze tracking mode results in reduced operating power requirements, as compared to the first gaze tracking mode.

[0037] Still further, the example AR device 10 includes an electronic display 22 that is selectively operable in either a “full display mode” or a “partial display mode”. The full display mode makes all image pixels available for generating an AR image 24, and the partial display mode makes only a subset of the image pixels available for generating the AR image 24. In other words, fewer than all image pixels provided by the electronic display 22 are used for generation of the AR image 24 during operation in the partial display mode. The image pixels that are not used for image generation during the partial display mode are referred to as “unused” image pixels or “unavailable” image pixels, with that latter reference alluding to the fact that such image pixels are not available for image generation during the partial display mode.

[0038] An advantageous attribute of the partial display mode of the electronic display 22 is that it includes or otherwise results in the powering down of at least a portion of display circuitry in the electronic display 22 that is associated with the image pixels that are unavailable— unused— in the partial display mode. For example, assume that the electronic display 22 has NxM image pixels arranged in a grid of N rows by M columns. In the full display mode, all N x M image pixels are available for generating the AR image 24. Contrastingly, in the partial display mode, fewer than NxM image pixels are available, e.g., one or more regions within the NxM grid are available for image generation, while the remaining image pixels are unavailable. Some or all of the display circuitry associated with the unavailable image pixels is powered down or otherwise placed in reduced power state, meaning that the partial display mode offers meaningful reductions in power consumption as compared to the full display mode.

[0039] Powering down or otherwise disabling of one or more portions of the display circuitry comprised in the electronic display 22 distinguishes from the kind of power reduction arising from, for example, simply increasing the extent of “dark” areas in an AR image. For example, with OLED displays in particular, image pixels corresponding to true black areas within an image may be off. However, the underlying circuitry associated with those image pixels— memory, addressing, and drive circuitry— generally remains active, thus limiting the achievable power savings.

[0040] Because the electronic display 22 outputs the AR image 24 as patterned light according to dynamically changing AR content, the term “AR image 24” does not refer to a static entity, and instead refers to whatever light is being output by the electronicP112065W001

[0041] display 22 at any given instant. Various display technologies maybe used, including light emitting diodes (LEDs) or micro-LEDs, organic LEDs (OLEDs) or micro-OLEDs, laser beam scanning (LBS) systems, etc. Thus, the mechanism for generating each “image pixel” differs as a function of the chosen display technology. Hence, the term “image pixel” therefore should be understood broadly as a defined or addressable point in an overall grid or array at which light output from the electronic display 22 can be controlled for image generation. For example, each image pixel is an addressable LED or OLED within a larger grid or may be a grid point within a larger scanning or raster grid used by an LBS system.

[0042] A light coupling mechanism 26 provides an optical path for coupling the AR image 24 to the viewing element 12, for displaying in the AR image display area 14 as a coupled AR image 28. Again, the image content being displayed to the wearer may change dynamically, such as in dependence on a changing field of view, as sensed by one or more camera systems 30 that are used to image the surrounding physical environment.

[0043] In one or more embodiments, a same configuration of the light coupling mechanism 26 provides coupling for the AR image 24, regardless of whether the electronic display 22 operates in the full display mode or in the partial display mode. In one or more other embodiments, the light coupling mechanism 26 is selectively operable in a first coupling mode or a second coupling mode. The first coupling mode supports or complements operation of the electronic display 22 in the full display mode, while the second coupling mode supports or complements operation of the electronic display 22 in the partial display mode.

[0044] The mode-selectable embodiments of the light coupling mechanism 26 complement variants of the partial display mode in which a contiguous strip or band of image pixels forms the subset of image pixels that are available for image generation. For example, an optical path of the light coupling mechanism 26 in the first coupling mode is configured to fill the AR image display area 14 with the AR image 24 as generated in the full display mode, where the full image pixel grid is available for image generation. Advantageously, the optical path in the second coupling mode is configured to fill the AR image display area 14 with the AR image 24 as generated in the partial display mode. For example, it splits or otherwise performs a distributed projection of the light emitted from a contiguous band of image pixels, to achieve the desired spatial separation between respective graphical elements displayed in the AR image display area 14.P112065W001

[0045] In this sense, “filling” the AR image display area 14 merely means that AR content— graphical or other visual elements— may extend to the boundaries of the AR image display area 14 in dependence on the actual AR content contained in the AR image 24 at any given instant. It does not mean that the AR content contained in the AR image 24 at any given instant necessarily occupies all of the AR image display area 14 or necessarily extends to the boundaries of the AR image display area 14.

[0046] Further components of the AR device 10 include device control circuitry 32 that is configured to determine a currently required gaze tracking resolution for tracking eye movements of the wearer of the AR device 10, and, responsive to the currently required gaze tracking resolution being satisfied by the second gaze tracking mode, select a reduced power mode of the AR device 10.

[0047] For example, the device control circuitry 32 switches the AR device 10 from a “normal” or “default” power mode that uses the first gaze tracking subsystem 18 for high resolution gaze tracking, to the reduced power mode, in response to determining that the gaze tracking resolution provided by the second gaze tracking subsystem 20 satisfies the currently required gaze tracking resolution. The terms “normal power mode” and “full power mode” are used interchangeably herein, and it should be appreciated that “full power mode” and “reduced power mode” are relative terms.

[0048] In particular, the AR device 10 has a characteristically lower average power consumption when operated in the reduced power mode as compared to the full power mode. Another way to define the terms “reduced power mode” and “full power mode” is to note that the circuitry used to provide the second gaze tracking resolution consumes less power than the circuitry used to provide the first gaze tracking resolution, and to note that the electronic display 22 consumes less power when operated in the partial display mode as compared to when operated in the full display mode.

[0049] With these potential power savings in mind, in one or more embodiments, the device control circuitry 32 is configured to periodically evaluate the currently required gaze tracking resolution and / or remain responsive to incoming signaling indicating the currently required gaze tracking resolution. In a case where the reduced power mode is already active and there is a new determination by the device control circuitry 32 that the second gaze tracking resolution satisfies the currently required gaze tracking resolution, the act of “selecting the reduced power mode” may be understood as reselecting or otherwise persisting the reduced power mode at least until a next evaluation of the currently required gaze tracking resolution. Thus, “selecting the reduced power mode” means switching from the full power mode to the reduced powerP112065W001

[0050] mode or deciding to remain in the reduced power mode rather than switching back to the full power mode.

[0051] Implementation of the device control circuitry 32 may vary in terms of the purpose and cost of the AR device 10. Broadly, the device control circuitry 32 comprises fixed circuitry or programmatically configured circuitry, or a mix of both. In at least one embodiment, the device control circuitry 32 comprises one or more microprocessors or other form of digital processing circuitry 34 that is programmatically configured to operate as the device control circuitry 32, based on the execution of computer program instructions (CPI) 36 that is held in storage 38.

[0052] The storage 38 in one or more embodiments also stores configuration and / or operational data 40. In an example case, the storage 38 comprises one or more types of computer readable media, such as nonvolatile memory for long term storage of computer programs and configuration data, and volatile memory for live operations, including program execution, maintaining working data, etc. Examples of nonvolatile memory include EEPROM, FLASH, or Solid State Disk (SSD), while examples of volatile memory include SRAM or DRAM.

[0053] However the device control circuitry 32 is implemented, it is configured to select the reduced power mode by selecting the second gaze tracking mode and selecting the partial display mode. In embodiments where the light coupling mechanism 26 has a first coupling mode for use with the full display mode and has a second coupling mode for use with the partial display mode, the device control circuitry 32 is further configured to select the second coupling mode of the light coupling mechanism 26, as part of selecting the reduced power mode of the AR device 10. That is, these various selections by the device control circuitry 32 and the resulting operational configuration of the AR device 10 constitute the reduced power mode.

[0054] Conversely, for operation of the AR device 10 in its normal power mode, the device control circuitry 32 is configured to select the first gaze tracking mode and select the full display mode. For embodiments that use a modal implementation of the light coupling mechanism 26 as described above, the full power mode further includes selection of the first coupling mode of the light coupling mechanism 26.

[0055] In one or more embodiments, the AR device 10 includes a host processing system 50, which comprises, for example, computer memory and one or more programmatically configured microprocessors. The host processing system 50 executes an operating system (OS) 52, according to which it provides a run-time environment (RTE) 54, for the execution of one or more software applications (APPs) 56. Such appsP112065W001

[0056] 56 may be games, business applications, multimedia applications, industrial applications, or essentially any type of application which makes use of the AR features provided by the AR device 10. Further, depending on its intended use, the AR device 10 in one or more embodiments includes one or more types of input / output (I / O) circuitry 58, such as control inputs, speakers, USB or other communication ports, etc.

[0057] In at least one embodiment, the OS 52 provides signaling to the device control circuitry 32, indicating the currently required gaze tracking resolution. Such signaling may originate from or depend on the app(s) 56 that are currently executing in the RTE 54, and it may indicate an actual resolution or it may provide a quantized indication such as “low resolution” or “high resolution.” As another example, the signaling maybe binary, such that a “zero” indication means that the second gaze tracking mode is sufficient or requested, while a “one” indication means that the first gaze tracking mode is needed or requested. This logic maybe reversed, of course. For example, the zero state maybe a default and it may map to selection of the full power mode as a default mode of operation of the AR device 10, with selective operation in the reduced power mode whenever the lower resolution of the second gaze tracking subsystem 20 is sufficient. Conversely, the reduced power mode maybe the default option. Regardless, in one or more embodiments, the device control circuitry 32 is configured to switch between the reduced power mode of the AR device 10 and a full or normal power mode of the AR device 10 on a dynamic basis, responsive to the signaling generated by the OS 52.

[0058] In at least one embodiment, the first gaze tracking subsystem 18 comprises a video-oculography (VOG) subsystem to provide the first gaze tracking resolution. For example, the VOG subsystem comprises one or more cameras, such as infrared cameras, that are positioned for imaging one or both eyes of the wearer of the AR device 10. The VOG subsystem may include image processors to determine eye movements and gaze direction from the camera images, or such processing may be performed by the device control circuitry 32.

[0059] In at least one embodiment, the second gaze tracking subsystem 20 comprises an electro-oculography (EOG) subsystem to provide the second gaze tracking resolution, which, as noted, is lower than the first gaze tracking resolution. For example, the EOG subsystem comprises one or more electrodes or skin contact points, for sensing electrical activity associated with eye movements of the wearer, along with the corresponding supporting analog and digital circuitry, for translating the detected electrical activity into gaze tracking information. Again, the associated signal processingP112065W001

[0060] may be performed by processing circuitry included in the EOG subsystem, or it may be performed by the device control circuitry 32.

[0061] In at least one embodiment, at least some graphical elements displayed in the AR image display area 14 while operating the AR device 10 in the reduced power mode have a greater minimum spatial separation, as compared to a minimum spatial separation used while operating the AR device 10 in the full power mode. For example, graphical control elements that are gaze selectable or gaze actuated may be displayed with a greater minimum separation during use of the second gaze tracking mode, as compared to the minimum separation applicable to use of the first gaze tracking mode.

[0062] Figure 2 illustrates an example arrangement of graphical elements 60— e.g., icons and / or selectable control elements— displayed as AR content within the AR image display area. In Figure 2, the depicted spatial separation of the graphical elements 60 necessitates or at least complements the use of the full power mode and its attendant higher gaze tracking resolution.

[0063] Figure 3 depicts an example arrangement of graphical elements 60 with increased spatial separation, complementing use of the reduced power mode, with its attendant lower gaze tracking resolution. One or more of the graphical elements 60 depicted in Figure 3 may be “next page” or “previous page” controls, reflecting the fact that the decreased control density may require multiple “pages” to display all controls or other information to the device wearer.

[0064] As one example, the minimum spatial separation applied while operating the AR device 10 in the reduced power mode is a function of the subset of available image pixels used in the partial display mode being divided into non-contiguous groups within an overall pixel grid of the electronic display 22. This point is understood better in the context of Figures 4-6.

[0065] Figure 4 depicts an overall defined grid of image pixels 70 provided by the electronic display 22, where all image pixels 70 of this “full grid” are available for generating the AR image 24 during the full display mode. Conversely, Figures 5 and 6 illustrate respective approaches to operation of the electronic display 22 in the partial display mode. Figure 5 represents a first variant of the partial display mode, with Figure 6 representing a second variant.

[0066] In both Figures 5 and 6, only a subset of image pixels 70 from the full grid is available for generating the AR image 24. Here and elsewhere, “subset” means “proper subset,” such that fewer than all image pixels 70 in the full grid are available for image generation in the partial display mode. Image pixels 70 that are not used in the partialP112065W001

[0067] display mode are referred to as unavailable image pixels 70 or unused image pixels 70. All or some of the supporting display circuitry that is associated with the unavailable image pixels 70 is powered down, for meaningful power savings during the partial display mode.

[0068] Figure 5 distinguishes from Figure 6 in that the available image pixels 70 are taken as a contiguous subset or region of image pixels 70 from the overall grid of image pixels 70. Particularly, the partial display variant illustrated by Figure 5 shows that a contiguous strip or band of image pixels 70 from the full grid is made available for image generation during the partial display mode, with image pixels 70 that are outside of that band being unused. Contrastingly, in the partial display variant shown in Figure 6, the subset of available image pixels 70 is itself divided into a plurality of smaller, noncontiguous subsets. Ideally, these smaller subsets are distributed over the full grid, so as to make use of the full area of the AR image display area 14 of the transparent viewing element 12.

[0069] One advantage of the approach taken in Figure 5 is simplification of the supporting display circuitry, or at least simplification of the controls needed to power down display circuitry associated with the unavailable image pixels 70. For example, in comparison to Figure 6, in Figure 5, one contiguous subset of pixel rows is active during the partial display mode, making it relatively simple to power down supporting circuitry— e.g., memory and drivers— associated with the other pixel rows. Indeed, as a general design proposition, powering down display circuitry associated with unused image pixels 70 is more straightforward with the approach taken in Figure 5 as compared to Figure 6.

[0070] On the other hand, the light coupling mechanism 26 needed to support the approach taken in Figure 5 may be more complicated as compared to the approach taken in Figure 6. To appreciate this point, consider that the distribution of available image pixel subsets over the full image pixel grid area as seen in Figure 6, along with the pixel distances separating those subsets, inherently provides the image area filling and spatial separation of graphical elements 60 needed or desired for operation in the second gaze tracking mode. Consequently, the same lensing or light coupling arrangement can be used for coupling the light emitted from the electronic display 22 operated in the full display mode suggested in Figure 4 and for coupling the light emitted from the electronic display 22 operated in the variant of the partial display mode suggested in Figure 6.P112065W001

[0071] However, such is not the case for the variant of the partial display mode suggested in Figure 5. There, a relatively small band or strip of contiguous image pixels 70 is available for image generation and the light coupling mechanism 26 must have an operational mode in which it distributes the light from the contiguous band of available image pixels 70, so as to fill the available AR image display area 14 of the transparent viewing element 12 and achieve the spatial separation of graphical elements 60 that is needed or required for operation with the second gaze tracking mode.

[0072] Figures 7-9 illustrate the foregoing distinctions, with Figure 7 introducing an example optical path 72 of the light coupling mechanism 26 in one or more embodiments and depicting the electronic display 22 being operated in the full display mode in which all image pixels 70 are available for generating the AR image 24— i.e., the entire pixel grid is usable for emitting light. In the depicted embodiment of the optical path 72, a first coupling mode of the light coupling mechanism 26 is selected for operation in the full display mode. Figure 8 illustrates that in the same embodiment, a second coupling mode of the light coupling mechanism 26 is selected for operation in the partial display mode and, in particular, in the contiguous-subset variant of the partial display mode shown in Figure 5.

[0073] With respect to the embodiment of the light coupling mechanism 26 depicted in Figures 7 and 8, the optical path 72 of the light coupling mechanism 26 comprises first and second lens systems 74 and 76, each having one or more lenses. The second lens system 76 is included in an optical path 72 of the light coupling mechanism 26 for operation in both the full power mode and the reduced power mode— i.e., in both the full display mode and the partial display mode. However, the first lens system 74 is included in the optical path 72 in series with the second lens system 76 for operation only in the reduced power mode— i.e., only when the partial display mode is active. This is because the first lens system 74 is configured to split light from the contiguous region or subset of available pixels 70 used in the partial display mode into a set of spatially separated beamlets for coupling into the second lens system 76. This beamlet separation results in the desired separation of graphical elements 60 and the corresponding filling or utilization of the AR image display area 14. As such, the first lens system 74 can be understood as providing a distributed projection of light from the contiguous subset of available pixels 70.

[0074] The first lens system 74 is rotatable or otherwise displaceable into and out of the optical path 72. Such movement maybe under direct or indirect control of the device control circuitry 32.P112065W001

[0075] Figure 9 illustrates the noncontiguous-subset variant of the partial display mode, as shown in Figure 6. Because spatial separation and image distribution is an inherent feature of dividing the available subset of image pixels 70 into smaller noncontiguous subsets spread out within the overall pixel grid, the beamlet separation features of the first lens system 74 is not needed. Thus, the optical path 72 need not be modal with respect to operation in the full display mode or the partial display mode.

[0076] In at least one embodiment corresponding to Figure 9, there is a separate part in the display controller that only can access the predefined areas, i.e., they are positioned in a sequence inside the display controller but are at different position in the active area of display. This selectively can be done by either have two control lines to the affected display pixels or via a switch in the display controller taking the control line from the affected pixels to two separate areas in the controller.

[0077] When running the electronic display 22 in partial mode, the clock frequency can be lowered and still maintain the same update frequency as in normal mode as there are fewer pixels that are going to be active. Another benefit of this is that the data that is going to be displayed can be positioned in an unbroken sequence in GRAM or other frame buffer that corresponds to the active partial display areas. This approach saves energy as storing data in consecutive positions in a GRAM / framebuffer can reduce energy consumption in comparison to having that data “spread out” in memory.

[0078] Particularly, having consecutive data makes several types of optimization available, such in terms of refresh cycles, sense amplifier complexity, and data access patterns.

[0079] Broadly, then, there are one or more embodiments where the light coupling mechanism 26 is selectively operable in a first coupling mode or a second coupling mode. An optical path 72 of the light coupling mechanism 26 in the first coupling mode is configured to fill the AR image display area with the AR image as generated in the full display mode, and the optical path 72 in the second coupling mode is configured to fill the AR image display area with the AR image as generated in the partial display mode. As such, selecting the reduced power mode of the AR device 10 includes selecting the second coupling mode of the light coupling mechanism 26, whereas selecting the full power mode includes selecting the first coupling mode of the light coupling mechanism 26.

[0080] Figure 10 illustrates a method 1000 of operating an AR device 10, according to one embodiment. The method 1000 includes determining (Block 1002) a currently required gaze tracking resolution for tracking eye movements of the wearer of the AR device 10. As described above, the AR device 10 supports a first gaze tracking modeP112065W001

[0081] having a first gaze tracking resolution and a second gaze tracking mode that requires less power than the first gaze tracking mode and has a second gaze tracking resolution that is lower than the first resolution.

[0082] The method 1000 further includes, responsive to the currently required gaze tracking resolution being satisfied by the second gaze tracking mode, selecting (Block 1004) a reduced power mode of the AR device 10. Selecting the reduced power mode comprises selecting the second gaze tracking mode and selecting a partial display mode of an electronic display used to generate an AR image displayed by the AR device. Here, the electronic display is selectively operable in a full display mode that makes all image pixels available for generating the AR image or in the partial display mode, which makes only a subset of the image pixels available for generating the AR image and powers down one or more portions of the electronic display associated with the image pixels that are unavailable.

[0083] In one or more embodiments of the method 1000, selecting the reduced power mode further comprises selecting a second coupling mode of a light coupling mechanism that is selectively operable in a first coupling mode or the second coupling mode. An optical path 72 of the light coupling mechanism in the first coupling mode is configured to fill an AR image display area 14 of the AR device 10 with the AR image 24 as generated in the full display mode, and the optical path 72 in the second coupling mode is configured to fill the AR image display area 14 with the AR image 24 as generated in the partial display mode. In such embodiments, selecting a full power mode of the AR device 10 comprises selection of the first gaze tracking mode, selection of the full display mode, and selection of the first coupling mode of the light coupling mechanism 26; and selecting the reduced power mode comprises selection of the second gaze tracking mode, selection of the partial display mode, and selection of the second coupling mode.

[0084] In one or more embodiments of the method 1000, determining the currently required gaze tracking resolution for tracking eye movements of the wearer of the AR device 10 comprises evaluating signaling generated by an OS 52 executing on a host processor 50 of the AR device 10.

[0085] In at least one embodiment, determining the currently required gaze tracking resolution for tracking eye movements of the wearer of the AR device 10 is performed on an ongoing basis, with the method 1000 further comprising dynamically selecting between a full power mode of the AR device 10 and the reduced power mode of the AR device 10, in dependence on changing gaze tracking resolution requirements. The fullP112065W001

[0086] power mode includes selection of the first gaze tracking mode and the reduced power mode includes selection of the second gaze tracking mode. In at least one such embodiment, selecting the partial display mode powers down at least one of: video buffers corresponding to image pixels that are unused in the partial display modes; or pixel addressing circuitry corresponding to the unused image pixels 70.

[0087] Figure 11 illustrates example details for the electronic display 22, according to one embodiment. A display controller 80 provides overall operational control of the electronic display 22 and provides for inter-processor communications. With momentary reference back to Figure 1, the inter-processor communications support mode control of the electronic display 22 by the device control circuitry 32, and image content / generation control by the host processing system 50.

[0088] One or more image data buffers 82— a type of memory circuit— holds image data for generation of the AR image 24, and pixel drive circuitry 84 provides corresponding on / off and color control of image pixels 70 as contained in or provided by one or more light emitting units 86. Power control circuitry 88 operates under control of the display controller 80 and initiates or causes the partial circuitry power down described earlier for the partial display mode. For example, during operation of the electronic display 22 in the partial display mode, any one or more applies: one or more image data buffers 82 associated with the unused image pixels 70 are powered down and / or portions of the pixel drive circuitry 84 associated with the unused image pixels 70 are powered down. Depending upon the nature of the light emitting unit(s) 86 associated with image pixels 70, the partial display mode may further include shutting down portions of the light emitting units 86 that are associated with the unused image pixels 70.

[0089] Among the various advantages provided by one or more of the foregoing embodiments is a meaningful reduction in the characteristic or average power consumption of an AR device 10, when high resolution gaze tracking is not required, with the concomitant advantage of allowing the AR device 10 to provide good battery life with a smaller battery pack. Smaller battery packs not only lower cost but improve user comfort in scenarios where the AR device 10 is worn or carried by its user.

[0090] A notable aspect is that the reductions in power consumption derive not only from the use of coarse or lower resolution gaze tracking in the reduced power mode, but also from the partial or selective powering down of display circuitry within the electronic display 22. Depending upon the particular approach taken for operating the electronic display 22 in the partial display mode, the AR device 10 may use a modal embodiment of a light coupling mechanism 26, where the mechanism operates inP112065W001

[0091] different modes in dependence on whether the electronic display 22 is operated in the full display mode or in the partial display mode.

[0092] In at least one embodiment, the optical path 72 of the light coupling mechanism uses a separate lens system between the light output by the electronic display 22 and the image viewed by the user. The approach may use one or more optical waveguides, but it is also applicable to “see-through” video applications. Further, the approach may be adapted for single or stereoscopic AR images 24.

[0093] The solution in one or more embodiments uses one or more double-sided microlens arrays (DSMLAs) that are positioned after the “normal” lens system but before the waveguide input. Here, the “normal” lens system is used in both the full display mode and the partial display mode. Selectively positioning the DSMLA(s) in or out of the optical path 72 may rely on mechanical positioning, or maybe electrically switched, e.g., such as an electric switch that controls a liquid lens type of activation. Figures 12 and 13 illustrate example arrangements of the optical path 72 of the light coupling mechanism 26 in one embodiment.

[0094] Figure 12 corresponds to the full display mode where all image pixels 70 are available for image generation and the optical path 72 includes a normal lens 90 that couples light— the AR image 24— output from the electronic display 22 into an optical waveguide 92, for displaying in the AR image display area 14 as the coupled AR image 28. Figure 13 corresponds to the partial display mode and, in particular, corresponds to the variant of the partial display mode in which a contiguous subset of image pixels 70 is used to form the AR image 24. The optical path 72 is reconfigured to include a DSMLA 94 that performs beamlet separation or distributed projection, for separation of the light from the contiguous subset of image pixels 70 into respective sub-images in the AR image display area 14.

[0095] With the DSMLA 94, a first microlens array facing the electronic display 22 is responsible for splitting the incoming light into multiple beamlets, while a second microlens array facing the waveguide 92 focuses these beamlets to the correct position, resulting in an array of smaller, separated images in the AR image display area 14. As noted, this arrangement maybe implemented for stereoscopic image generation and projection.

[0096] Using contiguous, full rows of image pixels 70 in the partial display mode means that the supporting circuitry, such as graphic random-access memory (GRAM) or other image data buffers, are used in an efficient way— i.e., image data can be written in an unbroken sequence. As a further variation, the partial display mode may use two orP112065W001

[0097] more distinct contiguous subsets of image pixels 70 and one or more DSMLAs may be used to achieve the desired distributed projection into the AR image display area 14.

[0098] In one or more such embodiments, the display controller 80, such as shown in Figure 11, is specially configured for addressing and managing those image pixels 70 that are used in the partial display mode. Effectively, the display controller 80 may therefore include control circuitry that is optimized for operation in the partial display mode, and this same logical and / or physical separation may extend to the image data buffers 82 and / or pixel drive circuitry 84, simplifying the powering down of the image pixels 70 that are unused in the partial display mode. Such optimizations may reduce the needed refresh cycles, lower the sense amplifier complexity, and simplify the data access patterns. However, the particular benefits depend on the overall type and architecture of the electronic display 22.

[0099] Notably, modifications and other embodiments of the disclosed invention(s) will come to mind to one skilled in the art having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. Therefore, it is to be understood that the invention(s) is / are not to be limited to the specific embodiments disclosed and that modifications and other embodiments are intended to be included within the scope of this disclosure. Although specific terms may be employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.

Claims

P112065W001CLAIMSClaims:

1. An augmented reality, AR, device (10) comprising:a transparent viewing element (12) positioned in front of one or both eyes of a wearer of the AR device, the transparent viewing element (12) including an AR image display area (14);a gaze tracking system (16) selectively operable in either a first gaze tracking mode having higher resolution or a second gaze tracking mode having lower power consumption; andan electronic display (22) selectively operable in either a full display mode or a partial display mode, the full display mode making all image pixels (70) available for generating an AR image (24), and the partial display mode making only a subset of the image pixels (70) available for generating the AR image (24) and powering down at least a portion of display circuitry (80, 82, 84, 86) in the electronic display (22) that is associated with the image pixels (70) that are unavailable in the partial display mode;a light coupling mechanism (26) configured to couple the AR image (24) to the AR image display area (14) of the transparent viewing element (12); and device control circuitry (32) configured to:determine a currently required gaze tracking resolution for tracking eye movements of the wearer of the AR device (10); and responsive to the currently required gaze tracking resolution being satisfied by the second gaze tracking mode, selecting a reduced power mode of the AR device (10) by: selecting the second gaze tracking mode and selecting the partial display mode.

2. The AR device (10) according to claim 1, wherein the light coupling mechanism (26) is selectively operable in a first coupling mode or a second coupling mode, wherein an optical path (72) of the light coupling mechanism (26) in the first coupling mode is configured to fill the AR image display area (14) with the AR image (24) as generated in the full display mode, and the optical path (72) in the second coupling mode is configured to fill the AR image display area (14) with the AR image (24) as generated in the partial display mode, and wherein selecting the reduced power mode of the AR device (10) further includes selecting the second coupling mode of the light coupling mechanism (26).Page 18 of 24P112065W0013. The AR device (10) according to claim 1 or 2, wherein the device control circuitry (32) is configured to determine the currently required gaze tracking resolution from signaling generated by an operating system (52) executing on a host processor (50) of the AR device.

4. The AR device (10) according to any one of claims 1-3, wherein the device control circuitry (32) is configured to switch between the reduced power mode of the AR device (10) and a full power mode of the AR device (10) on a dynamic basis, responsive to the signaling generated by the operating system (52), and wherein the full power mode comprises selection by the device control circuitry (32) of the first gaze tracking mode and the full display mode.

5. The AR device (10) according to any one of claims 1-4, wherein the gaze tracking system (16) comprises a video-oculography, VOG, subsystem (18) to provide a first gaze tracking resolution and further comprises an electro-oculography, EOG, subsystem (20) to provide a second gaze tracking resolution that is lower than the first gaze tracking resolution.

6. The AR device (10) according to claim 5, wherein graphical AR control elements (60) displayed in the AR image display area (14) while operating the AR device (10) in the reduced power mode have a greater minimum spatial separation, as compared to a minimum spatial separation used while operating the AR device (10) in the full power mode.

7. The AR device (10) according to claim 6, wherein the minimum spatial separation applied while operating the AR device (10) in the reduced power mode is a function of the subset of available image pixels (70) used in the partial display mode being divided into non-contiguous groups within an overall image pixel grid of the electronic display (22).

8. The AR device (10) according to claim 6, wherein the minimum spatial separation applied while operating the AR device (10) in the reduced power mode is provided by a distributed projection of light emitted from the subset of available imagePage 19 of 24P112065W001pixels (70) used in the partial display mode into the AR image display area (14) of the transparent viewing element (12), the distributed projection based on selecting a second coupling mode of the light coupling mechanism (26), as compared to a first coupling mode of the light coupling mechanism (26), which is selected for operation in the full power mode.

9. The AR device (10) according to any one of claim 1-8, wherein the light coupling mechanism (26) comprises first and second lens systems (74, 76), each having one or more lenses, wherein the second lens system (76) is included in an optical path (72) of the light coupling mechanism (26) for operation in both the full power mode and the reduced power mode, and wherein the first lens system (74) is included in the optical path (72) in series with the second lens system (76) for operation in the reduced power mode.

10. The AR device (10) according to claim 9, wherein the subset of available image pixels (70) used in the partial display mode is a contiguous region within an overall image pixel grid of the electronic display (22), and the first lens system (74) is configured to split light from the contiguous region into a set of spatially separated beamlets for coupling into the second lens system (76).

11. The AR device (10) according to claim 9 or 10, wherein the AR image (24) in the partial display mode maps respective graphical control elements (60) to respective beamlets.

12. The AR device (10) according to any one of claims 1-11, wherein the electronic display includes a display controller, and wherein the device control circuitry is configured to communicate with the display controller, for selection of the partial display mode or the full display mode.

13. The AR device (10) according to any one of claims 1-12, wherein the electronic display (22) comprises a laser beam scanner, LBS, that generates the AR image (24) by performing a raster scan using one or more laser beams, over an image pixel grid.Page 20 of 24P112065W00114. The AR device (10) according to any one of claims 1-12, wherein the electronic display (22) comprises one of a LCD, an LED display, an OLED display, or a micro-LED display.

15. The AR device (10) according to any one of claims 1-14, wherein a display controller (80) of the electronic display (22) is configured to power down one or more portions of display control circuitry (80, 82, 84, 86) for operation in the partial activation mode, the one or more portions corresponding to image pixels (70) that are unused in the partial display mode.

16. The AR device (10) according to claim 15, wherein the one or more portions of the display control circuitry (80, 82, 84, 86) that are powered down for operation in the partial activation mode comprise at least one of: video buffers (82) corresponding to the unused image pixels (70); or pixel addressing circuitry (84) corresponding to the unused image pixels (70).

17. A method (1000) of operation by an augmented reality, AR, device (10), the method (1000) comprising:determining (1002) a currently required gaze tracking resolution for tracking eye movements of the wearer of the AR device (10), wherein the AR device (10) supports a first gaze tracking mode having a first gaze tracking resolution and a second gaze tracking mode that requires less power than the first gaze tracking mode and has a second gaze tracking resolution that is lower than the first resolution; andresponsive to the currently required gaze tracking resolution being satisfied by the second gaze tracking mode, selecting (1004) a reduced power mode of the AR device (10) by:selecting the second gaze tracking mode; andselecting a partial display mode of an electronic display (22) used to generate an AR image (24) displayed by the AR device (10), the electronic display (22) selectively operable in a full display mode that makes all image pixels (70) available for generating the AR image (24) or in a partial display mode that makes only a subset of the image pixels (70) available for generating the AR image (24)Page 21 of 24P112065W001and powers down one or more portions of the electronic display (22) associated with the image pixels (70) that are unavailable.

18. The method (1000) according to claim 17, wherein selecting (1004) the reduced power mode further comprises selecting a second coupling mode of a light coupling mechanism (26) that is selectively operable in a first coupling mode or the second coupling mode, wherein an optical path (72) of the light coupling mechanism (26) in the first coupling mode is configured to fill an AR image display area (14) of the AR device (10) with the AR image (24) as generated in the full display mode, and the optical path (72) in the second coupling mode is configured to fill the AR image display area (14) with the AR image (24) as generated in the partial display mode.

19. The method (1000) according to claim 18, wherein selecting a full power mode of the AR device (10) comprises selection of the first gaze tracking mode, selection of the full display mode, and selection of the first coupling mode of the light coupling mechanism (26) and wherein selecting the reduced power mode comprises selection of the second gaze tracking mode, selection of the partial display mode, and selection of the second coupling mode.

20. The method (1000) according to any one of claims 17-19, wherein determining (1002) the currently required gaze tracking resolution for tracking eye movements of the wearer of the AR device (10) comprises evaluating signaling generated by an operating system (52) executing on a host processor (50) of the AR device (10).

21. The method (1000) according to any one of claims 17-20, wherein determining (1002) the currently required gaze tracking resolution for tracking eye movements of the wearer of the AR device (10) is performed on an ongoing basis, with the method (1000) further comprising dynamically selecting between a full power mode of the AR device (10) and the reduced power mode of the AR device (10), in dependence on changing gaze tracking resolution requirements, with the full power mode including selection of the first gaze tracking mode and the reduced power mode including selection of the second gaze tracking mode.

22. The method (1000) according to any one of claims 17-21, wherein a gaze tracking system (16) of the AR device (10) comprises a video-oculography, VOG, subsystem (18)Page 22 of 24P112065W001that provides the first gaze tracking resolution, and further comprises an electrooculography, EOG, subsystem (20) that provides the second gaze tracking resolution, and wherein selecting the second gaze tracking mode comprises activating the EOG subsystem (20) for gaze tracking.

23. The method (1000) according to any one of claims 17-22, wherein graphical AR control elements (60) displayed in the AR image display (24) while operating the AR device (10) in the reduced power mode have a greater minimum spatial separation within the AR display image area (14), as compared to a minimum spatial separation used while operating the AR device (10) in the full power mode.

24. The method (1000) according to any one of claims 17-23, wherein selecting the partial display mode powers down at least one of: video buffers (82) corresponding to image pixels (70) that are unused in the partial display modes; or pixel addressing circuitry (84) corresponding to the unused image pixels (70).Page 23 of 24

Images