Method of reducing the perceived motion-to-photon latency of an XR system

Abstract

Method of reducing the perceived motion-to-photon latency of an XR system A method of reducing the perceived motion-to-photon latency of an XR (extended reality) system, the method comprising driving an actuator assembly to move at least part of a display system of the XR system so as to reduce differences between a displayed image and a target image.

Description

[0001] P632-GB01 1

[0002] METHOD OF REDUCING THE PERCEIVED MOTION-TO-PHOTON LATENCY OF AN XR SYSTEM

[0003] Field

[0004] The present application relates to a method of reducing the perceived motion-to-photon latency of an XR (extended reality) system, the method comprising driving an actuator assembly to move at least part of a display system of the XR system so as to reduce differences between a displayed image and a target image.

[0005] Background

[0006] The present application relates to a method of reducing the perceived motion-to-photon latency of an XR (extended reality) system.

[0007] In head-mounted displays the images projected by the display need to move with the world around the user when the user turns their head.

[0008] Excessive latency between motion and photon is undesirable. In a public paper DAGRI (see https: / / medium.eom / @DAQRI / motion-to-photon-latency-in-mobile-ar-and-yr-99f82c480926), an estimate the acceptable latency is considered to be less than 10ms for virtual reality (VR) systems and less than 5ms for Augmented Reality (AR) systems.

[0009] The native latency of a system that gathers position from a camera, calculates the desired display image and then projects that image from a display is much larger than this. The latency of the position calculation can be reduced by using gyroscopes and accelerometers to interpolate motion between camera frames. It this then possible to predict the motion from the recent motion history (velocity and acceleration) and use this to calculate a future position. The image that is 3D rendered can rendered so that it is correct for that anticipated position. Limited further corrections can be made through what is known as 2D "late warping" (also called time-warping) while the image is being displayed to partially correct for corrections to the initial prediction.

[0010] The limitations of this approach are the computational weight, the errors in the position prediction and the limited corrections that can be made through late warping.

[0011] Summary

[0012] According to an aspect of the present invention, there is provided a method of reducing the perceived motion-to-photon latency of an XR (extended reality) system, the method comprising: determining a predicted pose (i.e. position and orientation) of an XR device at a future point in time, based on dataP632-GB01 2

[0013] provided on the pose and motion (e.g. velocity, and / or acceleration) of the XR device by one or more sensors of the XR system; preparing at least part of an image (e.g. a line of an image, or a sub-frame for a colour component of an image) based on a notional pose of the XR device determined at least in part based on the predicted pose of the XR device at the future point in time; displaying the prepared at least part of an image on a display system of the XR system at the future point in time; driving an actuator assembly to move at least part of the display system so as to reduce (e.g. to compensate for, or to mitigate) any differences between the displayed at least part of an image, and at least part of a target image that would have been displayed if the image preparing had been based on the pose of the XR device at the future point in time (instead of the notional pose).

[0014] The method may remove the perceived motion-to-photon latency of the XR system. By using the display, this removes or reduces the need for late warping and so removing the disadvantages associated with that. This may also provide an improved experience for the user.

[0015] The XR device may be a head mounted device. The display system of the XR system may form part of the XR device. The display system may be a display screen, or an optical component through which a display screen is viewed. The actuator assembly may drive movement of the at least part of the display system relative to a support structure of the display system, and / or relative to the eyes of the user of the display system.

[0016] The target image may be referred to as an 'ideal image' or as a 'zero-latency image'.

[0017] Optionally, driving the actuator assembly comprises powering an SMA (shape memory alloy) element configured, when powered, to (directly or indirectly) apply an actuating force to the at least part of the display system capable of moving the at least part of the display system relative to a support structure of the display system (and e.g. relative to the eyes of the user of the display system).

[0018] Optionally, the notional pose of the XR device is determined based on: (i) the predicted pose of the XR device at the future point in time, and (ii) a determined position (which may be measured or calculated) or a predicted (future) position of the actuator assembly within a range of movement.

[0019] The position of the actuator assembly may be the position of the at least part of the display system relative to the support structure of the display system.

[0020] Optionally, if the determined or predicted position of the actuator assembly corresponds to a position at an end of the range of movement, the notional pose is determined as a pose that is expected to causeP632-GB01 3

[0021] the actuator assembly to be moved towards the centre of the range of movement when the actuator assembly is driven to reduce any differences between the displayed at least part of an image, and the at least part of a target image.

[0022] The predicted position of the actuator assembly may be a position the actuator assembly is predicted to be in, within the range of movement, at the future point in time.

[0023] Optionally, the notional pose of the XR device is determined further based on the velocity and / or acceleration capabilities of the actuator assembly.

[0024] Optionally, the image preparing comprises rendering at least part of an image based on a predicted pose of the XR device at the future point in time determined pre rendering. Where this is the case, the notional pose may correspond to the pre-render predicted pose of the XR device; and the prepared at least part of an image may correspond to the rendered at least part of an image.

[0025] Optionally, the image preparing comprises rendering at least part of an image based on an adjusted predicted pose of the XR device determined pre rendering, wherein the adjusted predicted pose is determined based on: (i) a predicted pose of the XR device at the future point in time determined pre rendering, and (ii) a determined position (which may be measured or calculated) or a predicted (future) position of the actuator assembly (within a range of movement) determined pre rendering. Where this is the case, the notional pose may correspond to the pre-render adjusted predicted pose of the XR device; and the prepared at least part of an image may correspond to the rendered at least part of an image.

[0026] Optionally, the image preparing comprises late-warping (also known as time-warping) the rendered at least part of an image based on a (e.g. measured or calculated) pose of the XR device determined post rendering. Where this is the case, the notional pose may correspond to the determined pose of the XR device; and the prepared at least part of an image may correspond to the late-warped at least part of an image.

[0027] Optionally, the image preparing comprises late-warping the rendered at least part of an image based on an adjusted determined (e.g. measured or calculated) pose of the XR device, wherein the adjusted determined pose is determined based on: (i) a (e.g. measured or calculated) pose of the XR device determined post rendering, and (ii) a determined position (which may be measured or calculated) or a predicted (future) position of the actuator assembly (within a range of movement) determined post rendering. Where this is the case, the notional pose may correspond to the adjusted determined pose ofP632-GB01 4

[0028] the XR device; and the prepared at least part of an image may correspond to the late-warped at least part of an image.

[0029] Optionally, the image preparing comprises late-warping the rendered at least part of an image based on a predicted pose of the XR device at the future point in time determined post rendering. Where this is the case, the notional pose may correspond to the post-render predicted pose of the XR device; and the prepared at least part of an image may correspond to the late-warped at least part of an image.

[0030] Optionally, the image preparing comprises late-warping the rendered at least part of an image based on an adjusted predicted pose of the XR device determined post rendering, wherein the adjusted predicted pose is based on: (i) a predicted pose of the XR device at the future point in time determined post rendering, and (ii) a determined position (which may be measured or calculated) or a predicted (future) position of the actuator assembly (within a range of movement) determined post rendering. Where this is the case, the notional pose may correspond to the post-render adjusted predicted pose of the XR device; and the prepared at least part of an image may correspond to the late-warped at least part of an image.

[0031] Optionally, the at least part of the display system is moved to compensate for any differences between the notional pose (used for the image preparing) and the pose of the XR device at the future point in time.

[0032] Optionally, the prepared at least part of an image is a line of an image; wherein the display system is a line sequential display system; and wherein the method comprises carrying out the steps of predicting a future pose, image preparing, image displaying, and driving the actuator assembly, for each line of the line sequential display system.

[0033] Optionally, the prepared at least part of an image is a sub-frame for a colour component of an image; wherein the display system is a colour sequential display system; and wherein the method comprises carrying out the steps of predicting a future pose, image preparing, image displaying, and driving the actuator assembly, for each colour component of the colour sequential display system.

[0034] Optionally, the prepared at least part of an image is a complete image (frame), and wherein the display system is a simultaneous display system.

[0035] Optionally, the method comprises moving the at least part of the display system so as to compensate for any motion (e.g. any change in pose) of the XR device (e.g. only) during the image displaying.P632-GB01 5

[0036] Optional ly, the method is a method of reducing the perceived motion-to-photon latency of an XR system to 20 ms or less, to 10 ms or less, or to 5 ms or less.

[0037] Optionally, the one or more sensors comprises at least one visual sensor (e.g. camera) and at least one inertial sensor (e.g. accelerometer or gyroscope); and wherein the determining of a predicted pose of the XR device at a future point in time comprises using data provided by the at least one visual sensor and the at least one inertial sensor.

[0038] The post-render determined pose of the XR device may be determined using data provided by the at least one inertial sensor.

[0039] Inertial sensors may also be referred to as inertial measurement units (IMUs).

[0040] According to another aspect of the present invention, there is provided a (e.g. non-transitory) computer-readable medium storing (program) instructions for causing a system to perform the method described above.

[0041] According to another aspect of the present invention, there is provided an XR (extended reality) system comprising: an XR device; one or more sensors configured to provide data on the pose and motion of the XR device (and thus also the position and motion of the user of the XR device); a predictor configured to determine a predicted pose of the XR device at a future point in time, based on data provided by the one or more sensors; an image processor configured to prepare at least part of an image based on a notional pose of the XR device determined at least in part based on the predicted pose of the XR device at the future point in time; a display system configured to display (to the user of the XR device) the prepared image (e.g. on a display screen of the display system) at the future point in time; and an actuator assembly configured to move at least part of the display system so as to reduce the perceived motion-to-photon latency of the XR system.

[0042] The XR device may be a head mounted device. The at least part of the display system may be a display screen, or an optical component through which the display screen is viewed. The actuator assembly may be configured to move the at least part of the display system relative to a support structure of the display system, and / or relative to the eyes of the user of the display system.

[0043] Optionally, the actuator assembly comprises an SMA (shape memory alloy) element configured, when powered, to (directly or indirectly) apply an actuating force to the at least part of the display systemP632-GB01 6

[0044] capable of moving the at least part of the display system relative to a support structure of the display system (and e.g. relative to the eyes of the user of the display system).

[0045] Optionally, the image processor is configured to render at least part of an image (e.g. a line of an image, or a sub-frame for a colour component of an image).

[0046] Optionally, the image processor is configured to late-warp the rendered at least part of an image.

[0047] Optionally, the one or more sensors comprises at least one visual sensor (e.g. camera) and at least one inertial sensor (e.g. accelerometer and / or gyroscope).

[0048] Optionally, the XR system comprises the above-mentioned computer-readable medium.

[0049] Brief description of the drawings

[0050] Certain embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings in which:

[0051] Figure 1 is a flowchart showing the steps involved in a method of reducing the perceived motion-to- photon latency of an XR system;

[0052] Figure 2 is a flowchart showing sub-steps of step S20 of Fig. 1;

[0053] Figure 3 is a graph illustrating how the method of Fig. 1 may be used with a line sequential display; Figure 4 is a graph illustrating how the method of Fig. 1 may be used with a colour sequential display; Figure 5 is a block diagram illustrating an XR system.

[0054] Detailed description

[0055] Method of reducing perceived photon-to-motion latency of an XR system

[0056] Fig. 1 is a flowchart showing the steps involved in a method of reducing and / or removing the perceived motion-to-photon latency of an XR (extended reality) system 110. This method may reduce the perceived motion-to-photon latency of an XR system 100 to 20 ms or less, to 10 ms or less, or to 5 ms or less.

[0057] Firstly, the method involves determining a predicted pose (i.e. position and orientation) of an XR device 110, attached or held by the user of the XR system 100, at a future point in time, based on data provided on the pose and motion (e.g. velocity, and / or acceleration) of the XR device 110 by one or more sensors 111 of the XR system S10.P632-GB01 7

[0058] The XR device 110 may be a head-mounted device (HMD) of the XR system 100. For example, the XR device 110 may be a VR (virtual reality) headset, an AR (augmented reality) headset, AR glasses, or a MR (mixed reality) headset. The HMD may be a standalone HMD which has built-in computing power, and does not require an external PC or external hardware, or may be a tethered HMD. The XR device 110 may alternatively be an input device of the XR system 100 configured to allow the user of the XR system 100 to interact with the XR environment.

[0059] As shown in Fig. 5, the one or more sensors 111 may form part of the XR device 110 for inside-out tracking of the movement of the XR device 110. Additionally, or alternatively, the one or more sensors 111 may form part of an external device of the XR system 100 for outside-in tracking.

[0060] The one or more sensors 111 may comprise at least one visual sensor (e.g. at least one camera) and at least one inertial sensor (e.g. at least one accelerometer or gyroscope), and the determining of a predicted pose of the XR device 110 at a future point in time may involve using data provided by the at least one visual sensor and the at least one inertial sensor. Both the visual and inertial sensors may form part of the XR device 110, but this may not necessarily be the case. For example, the XR device 110 could be provided with the at least one inertial sensor, and an external device of the XR system 100 could be provided with the at least one visual sensor. Inertial sensors may also be referred to as inertial measurement units (IMUs).

[0061] The method further involves preparing at least part of an image based on a notional pose of the XR device 110 determined at least in part based on the predicted pose of the XR device 110 at the future point in time S20.

[0062] The at least part of an image may be a line of an image, a sub-frame for a colour component of an image, or a complete image frame.

[0063] The method further involves displaying the prepared at least part of an image on a display system 114 of the XR system 100 at the future point in time S30.

[0064] The display system 114 of the XR system 100 may form part of the XR device 110 e.g. wherein the XR device 110 is an HMD. The display system 114 may be a display screen, or an optical component through which a display screen is viewed.

[0065] The method further involves driving an actuator assembly 115 to move at least part of the display system 114 so as to reduce (e.g. to compensate for, or to mitigate) any differences between theP632-GB01 8

[0066] displayed at least part of an image, and at least part of a target image that would have been displayed if the image preparing had been based on the actual pose of the XR device 110 at the future point in time (instead of the notional pose) S40. In other words, the at least part of the display system 114 is moved by the actuator assembly 115 to compensate for any differences between the notional pose and the pose of the XR device 110 at the future point in time.

[0067] The actuator assembly 115 may drive movement of the at least part of the display system 114 relative to a support structure of the display system 114, and e.g. relative to the eyes of the user of the display system 114.

[0068] The target image may be referred to as an 'ideal image' or as a 'zero-latency image'.

[0069] Driving the actuator assembly 115 may comprise powering an SMA (shape memory alloy) element configured, when powered, to apply an actuating force to the at least part of the display system 114 capable of moving the at least part of the display system 114 relative to a support structure of the display system 114 and e.g. relative to the eyes of the user of the display system 114. The SMA element may be configured to, directly or indirectly, apply said actuating force to the at least part of the display system 114.

[0070] The notional pose of the XR device 110 of step S20 may correspond to the predicted pose of the XR device 110 at the future point in time of step S20.

[0071] Alternatively, the notional pose of the XR device 110 may be determined based on: (i) the predicted pose of the XR device 110 at the future point in time, and (ii) a determined position (which may be measured or calculated) or a predicted (future) position of the actuator assembly 115 within a range of movement. For example, if the determined or predicted position of the actuator assembly 115 corresponds to a position at an end of the range of movement, the notional pose may be determined as a pose that is expected to cause the actuator assembly 115 to be moved towards the centre of the range of movement when the actuator assembly 115 is driven to reduce any differences between the displayed at least part of an image, and the at least part of a target image.

[0072] Moreover, the notional pose of the XR device 110 may be determined further based on the velocity and / or acceleration capabilities of the actuator assembly 115.

[0073] Fig.2 is a flowchart showing sub-steps of step S20 relating to preparing the at least part of an image based on a notional pose of the XR device 110.P632-GB01 9

[0074] The image preparing step S20 may comprise rendering at least part of an image based on a predicted pose of the XR device 110 at the future point in time determined pre rendering S21A. Sub-step S21A may be the only sub-step of step S20, in which case the notional pose of step S20 would correspond to the pre-render predicted pose of the XR device 110 of sub-step S21A, and the prepared at least part of an image of step S30 would correspond to the rendered at least part of an image of sub-step S21A.

[0075] Alternatively, the image preparing step S20 may comprise rendering at least part of an image based on an adjusted predicted pose of the XR device 110 determined pre rendering S21B, wherein the adjusted predicted pose is determined based on: (i) a predicted pose of the XR device 110 at the future point in time determined pre rendering, and (ii) a determined position ora predicted (future) position of the actuator assembly 115, within a range of movement, determined pre rendering. This adjusted predicted pose corresponds to the predicted pose of the XR device 110 determined pre rendering but adjusted to try to cause the actuator assembly 115 to be moved towards the centre of the range of movement when the actuator assembly 115 is driven to reduce any differences between the displayed at least part of an image and the at least part of a target image. Sub-step S21B may be the only sub-step of step S20, in which case the notional pose of step S20 would correspond to the pre-render adjusted predicted pose of the XR device 110 of sub-step S21B, and the prepared at least part of an image of step S30 would correspond to the rendered at least part of an image of sub-step S21B.

[0076] It is noted that the determined position of the actuator assembly 115 may be a measured or calculated position of the actuator assembly 115 within the range of movement. The position of the actuator assembly 115 may refer to the position of the at least part of the display system 114 relative to the support structure of the display system 114. Moreover, the range of movement of the actuator assembly 115 may refer to the limited range of movement the at least part of the display system 114 is capable of being moved within, relative to a support structure of the display system 114. It is further noted that the predicted position of the actuator assembly 115 may be a position the actuator assembly 115 is predicted to be in, within the range of movement, at the future point in time.

[0077] In addition to the rendering sub-step S21, the image preparing step S20 may comprise a further substep S22 involving late-warping (also known as time-warping) the rendered at least part of an image.

[0078] The late-warping may be based on a (e.g. measured or calculated) pose of the XR device 110 determined post rendering S22A. Where the image preparing step S20 further comprises this sub-step S22A, the notional pose of step S20 would correspond to the determined pose of the XR device 110 of sub-stepP632-GB01 10

[0079] S22A, and the prepared at least part of an image of step S30 would correspond to the late-warped at least part of an image of sub-step S22A.

[0080] Alternatively, the late-warping may be based on an adjusted determined pose of the XR device S22B, wherein the adjusted determined pose is determined based on: (i) a (e.g. measured or calculated) pose of the XR device 110 determined post rendering, and (ii) a determined position (which may be measured or calculated) or a predicted (future) position of the actuator assembly 115, within a range of movement, determined post rendering. The adjusted determined pose corresponds to the pose of the XR device 110 determined post rendering but adjusted to try to cause the actuator assembly 115 to be moved towards the centre of the range of movement when the actuator assembly 115 is driven to reduce any differences between the displayed at least part of an image and the at least part of a target image. Where the image preparing step S20 further comprises this sub-step S22B, the notional pose of step S20 would correspond to the adjusted determined pose of the XR device 110 of sub-step S22B, and the prepared at least part of an image of step S30 would correspond to the late-warped at least part of an image of sub-step S22B.

[0081] Alternatively, the late-warping may be based on a predicted pose of the XR device 110 at the future point in time determined post rendering S22C. Where the image preparing step S20 further comprises this sub-step S22C, the notional pose of step S20 would correspond to the post-render predicted pose of the XR device 110 of sub-step S22C, and the prepared at least part of an image of step S30 would correspond to the late-warped at least part of an image of sub-step S22C.

[0082] Alternatively, the late-warping may be based on an adjusted predicted pose of the XR device 110 determined post rendering S22D, wherein the adjusted predicted pose is based on: (i) a predicted pose of the XR device 110 at the future point in time determined post rendering, and (ii) a determined position (which may be measured or calculated) or a predicted (future) position of the actuator assembly 115, within a range of movement, determined post rendering. This adjusted predicted pose corresponds to the predicted pose of the XR device 110 determined post rendering but adjusted to try to cause the actuator assembly 115 to be moved towards the centre of the range of movement when the actuator assembly 115 is driven to reduce any differences between the displayed at least part of an image and the at least part of a target image. Where the image preparing step S20 further comprises this sub-step S22D, the notional pose of step S20 would correspond to the post-render adjusted predicted pose of the XR device 110 of sub-step S22D, and the prepared at least part of an image of step S30 would correspond to the late-warped at least part of an image of sub-step S22D.P632-GB01 11

[0083] In sub-steps S22A and S22B, the post-render determined pose of the XR device 110 may be determined using data provided by the at least one inertial sensor.

[0084] Line sequential display

[0085] The prepared at least part of an image may be a line of an image, and the display system 114 may be a line sequential display system (e.g. an OLED display system). Where this is the case, the steps of predicting a future pose S10, image preparing S20, image displaying S30, and driving the actuator assembly S40 may be carried out for each line of the line sequential display system.

[0086] Fig. 3 is a graph illustrating how the method of Fig. 1 may be used with a line sequential display system.

[0087] Fig. 3 shows a pose P versus time T graph. A first frame is displayed between times T2 and T3 (i.e. period DI), a second frame is displayed between times T3 and T4 (i.e. period D2), and a third frame is displayed between times T4 and T5 (i.e. period D3). For each frame, the steps of predicting a future pose S10, image preparing S20, image displaying S30, and driving the actuator assembly S40 are carried out for each line of the line sequential display system. The image preparing S20 in this example only involves the rendering sub-step S21B. The rendering of the first, second, and third frames are respectively carried out between times T1 and T2, T2 and T3, and T3 and T4 (i.e. periods Rl, R2, and R3). The predicting of future poses S10 is carried out for the first, second, and third frames between times TO and Tl, T1 and T2, and T2 and T3, respectively.

[0088] Line AP depicts the movement of the XR device. This movement does not involve any changes in orientation, just changes in position, in this example. Line NP depicts the notional poses of the XR device that are used for the image preparing S20 for each line of the line sequential display. These notional poses correspond to the 'adjusted predicted poses' of the XR device that are used for each rendering sub-step S21B. Line PP depicts the 'predicted poses' of the XR device that are used for each rendering sub-step S21B. Line AA depicts the movement driven by the actuator assembly.

[0089] As shown in Fig. 3, when the first frame is displayed during period DI, there is a separation between lines AP and NP representing discrepancies between actual poses of the XR device and notional poses of the XR device. These discrepancies are compensated by the actuator assembly as illustrated by the line AA between times T2 and T3. The separation between lines AP and NP grows during period D2 when the second frame is displayed. This further separation is also compensated by the actuator assembly as illustrated by the line AA between times T3 and T4.P632-GB01 12

[0090] When the third frame is displayed during period D3, there is a separation between the lines NP and PP. This separation is due to adjustments made to the notional poses used for the third frame, which have been made to try to cause the actuator assembly to be moved back towards the centre of its range of movement, as part of sub-step S21B. The reduction in the separation of the line AA from the x-axis in this period illustrates that the separation between lines NP and PP has been effective in causing the actuator assembly to be moved back towards the center of its range of movement.

[0091] Colour sequential display

[0092] The prepared at least part of an image may be a sub-frame for a colour component of an image, and the display system 114 may be a colour sequential display system (e.g. an LCOS or pLED display system). Where this is the case, the steps of predicting a future pose S10, image preparing S20, image displaying S30, and driving the actuator assembly S40 may be carried out for each colour component of the colour sequential display system.

[0093] Fig. 4 is a graph illustrating how the method of Fig. 1 may be used with a colour sequential display.

[0094] Fig. 4 shows a pose P versus time T graph. A first frame is displayed between times T2 and T3 (i.e. period DI), a second frame is displayed between times T3 and T4 (i.e. period D2), and a third frame is displayed between times T4 and T5 (i.e. period D3). Each frame is split into three sub-frames for each colour component forming the frame, i.e. each frame is split into red, green, and blue colour components. These sub-frames are displayed one after the other during periods RS, GS, and BS, which are separated by blanking periods. For each frame, the steps of predicting a future pose S10, image preparing S20, image displaying S30, and driving the actuator assembly S40 are carried out for each subframe. The image preparing S20 in this example only involves the rendering sub-step S21B. The rendering of the first, second, and third frames are respectively carried out between times T1 and T2, T2 and T3, and T3 and T4 (i.e. periods Rl, R2, and R3). The predicting of future poses S10 is carried out for the first, second, and third frames between times TO and Tl, T1 and T2, and T2 and T3, respectively.

[0095] Like with Fig. 3, line AP depicts the movement of the XR device. This movement does not involve any changes in orientation, just changes in position, in this example. Line NP depicts the notional poses of the XR device that are used for the image preparing S20 for each sub-frame. These notional poses correspond to the 'adjusted predicted poses' of the XR device that are used for each rendering sub-step S21B. Line PP depicts the 'predicted poses' of the XR device that are used for each rendering sub-step S21B. Line AA depicts the movement driven by the actuator assembly.P632-GB01 13

[0096] As shown in Fig. 4, when the first frame is displayed during period DI, there is a separation between lines AP and NP representing discrepancies between actual poses of the XR device and notional poses of the XR device. These discrepancies are compensated by the actuator assembly as illustrated by the line AA between times T2 and T3. The separation between lines AP and NP grows during period D2 when the second frame is displayed. This further separation is also compensated by the actuator assembly as illustrated by the line AA between times T3 and T4.

[0097] When the third frame is displayed during period D3, there is a separation between the lines NP and PP. This separation is due to adjustments made to the notional poses used for the third frame, which have been made to try to cause the actuator assembly to be moved back towards the centre of its range of movement, as part of sub-step S21B. The reduction in the separation of the line AA from the x-axis in this period illustrates that the separation between lines NP and PP has been effective in causing the actuator assembly to be moved back towards the center of its range of movement.

[0098] As shown in Fig. 4, the actuator assembly is configured to compensate for discrepancies between the actual pose of the XR device and the notional pose of the XR device when the display is active, but not necessarily during blanking periods. Moreover, the actuator assembly is configured such that it moves back towards the centre of its range of movement (i.e. towards the x-axis) during blanking periods whenever reasonable.

[0099] Simultaneous display

[0100] The prepared at least part of an image may be a complete image (i.e. a full image frame) and the display system 114 may be a simultaneous display system (e.g. a pLED display system). Where this is the case, the steps of predicting a future pose S10, image preparing S20, image displaying S30, and driving the actuator assembly S40 may be carried out for each frame of the simultaneous display system.

[0101] As with colour sequential displays, simultaneous displays may have blanking periods. Where this is the case, the actuator assembly may be configured to compensate for discrepancies between the actual pose of the XR device and the notional pose of the XR device when the display is active, but not necessarily during blanking periods. Moreover, the actuator assembly may be configured to move back towards the centre of its range of movement during blanking periods whenever reasonable.

[0102] Computer-readable medium

[0103] There may be provided a (e.g. non-transitory) computer-readable medium 116 storing (program) instructions for causing a system to perform the above-described method of reducing perceived photon-to-motion latency.P632-GB01 14

[0104] XR system

[0105] Fig.5 is a block diagram illustrating the XR system 110. The XR (extended reality) system 100 comprises: the XR device 110; the one or more sensors 111 configured to provide data on the pose and motion of the XR device 110 (and thus also the position and motion of the user of the XR device 110); a predictor 112 configured to determine a predicted pose of the XR device 110 at a future point in time, based on data provided by the one or more sensors 111; an image processor 113 configured to prepare at least part of an image based on the notional pose of the XR device 110; the display system 114 configured to display (to the user of the XR device 110) the prepared image (e.g. on a display screen of the display system 114) at the future point in time; and the actuator assembly 115 configured to move at least part of the display system 114 so as to reduce the perceived motion-to-photon latency of the XR system 100.

[0106] As discussed above, the XR device 110 may be a head mounted device. The at least part of the display system 114 may be a display screen, or an optical component through which the display screen is viewed. The actuator assembly 115 may be configured to move the at least part of the display system 114 relative to a support structure of the display system 114, and / or relative to the eyes of the user of the display system 114.

[0107] As discussed above, the actuator assembly 115 may comprise an SMA (shape memory alloy) element configured, when powered, to apply an actuating force to the at least part of the display system 114 capable of moving the at least part of the display system 114 relative to a support structure of the display system 114, and e.g. relative to the eyes of the user of the display system 114.

[0108] The image processor 113 may be configured to carry out the rendering of at least part of an image. The image processor 113 may also be configured to late-warp the rendered at least part of an image.

[0109] As discussed above, the one or more sensors 111 may comprise at least one visual sensor (e.g. camera) and at least one inertial sensor (e.g. accelerometer and / or gyroscope).

[0110] The XR system 100 may comprise the above-mentioned computer-readable medium 116.

[0111] Actuator assemblies

[0112] In some examples, as disclosed in WO 2013 / 175197 Al which is incorporated herein by reference to the maximum extent permissible by law, the actuator assembly 115 may comprise four actuating units arranged so as to be capable of moving the at least part of the display system in any direction in a movement plane without applying any net torque to the at least part of the display system about a firstP632-GB01 15

[0113] axis perpendicular to the movement plane. A first pair of the four actuating units may each be configured to apply a torque to the at least part of the display system in one sense about the first axis, and a second pair of the four actuating units may each be configured to apply a torque to the at least part of the display system in the other sense about the first axis.

[0114] In some examples, the actuator assembly 115 may comprise a plurality of actuating units arranged such that, for each direction along each axis of a Cartesian coordinate system, there is at least one actuating force with a non-zero component along that direction, as e.g. disclosed in WO 2011 / 104518 Al which is incorporated herein by reference to the maximum extent permissible by law.

[0115] Many other actuator arrangements may be suitable for the actuator assembly 115 of the XR device 110.

[0116] Other variations

[0117] It will be appreciated that there may be many other variations of the above-described examples.

[0118] For example, it will be appreciated that the XR device 110 may be different to what is shown in Fig. 5. For example, the XR device 110 may not comprise the one or more sensors 111, as these may be provided as part of an external device of the XR system 100, e.g. for outside-in tracking.

[0119] SMA

[0120] The term 'shape memory alloy (SMA) element' may refer to any element comprising SMA. The SMA element may be described as an SMA wire. The SMA element may have any shape that is suitable for the purposes described herein. The SMA element may be elongate and may have a round cross section or any other shape cross section. The cross section may vary along the length of the SMA element. The SMA element might have a relatively complex shape such as a helical spring. It is also possible that the length of the SMA element (however defined) may be similar to one or more of its other dimensions. The SMA element may be sheet-like, and such a sheet may be planar or non-planar. The SMA element may be pliant or, in other words, flexible. In some examples, when connected in a straight line between two components, the SMA element can apply only a tensile force which urges the two components together. In other examples, the SMA element may be bent around a component and can apply a force to the component as the SMA element tends to straighten under tension. The SMA element may be beam-like or rigid and may be able to apply different (e.g. non-tensile) forces to elements. The SMA element may or may not include material(s) and / or component(s) that are not SMA. For example, the SMA element may comprise a core of SMA and a coating of non-SMA material. Unless the context requires otherwise, the term 'SMA element' may refer to any configuration of SMA material acting as aP632-GB01 16

[0121] single actuating element which, for example, can be individually controlled to produce a force on an element. For example, the SMA element may comprise two or more portions of SMA material that are arranged mechanically in parallel and / or in series. In some arrangements, the SMA element may be part of a larger SMA element. Such a larger SMA element might comprise two or more parts that are individually controllable, thereby forming two or more SMA elements. The SMA element may comprise an SMA wire, SMA foil, SMA film or any other configuration of SMA material. The SMA element may be manufactured using any suitable method, for example by a method involving drawing, rolling, deposition, sintering or powder fusion. The SMA element may exhibit any shape memory effect, e.g. a thermal shape memory effect or a magnetic shape memory effect, and may be controlled in any suitable way, e.g. by Joule heating, another heating technique or by applying a magnetic field.

Claims

P632-GB01 17Claims1. A method of reducing the perceived motion-to-photon latency of an XR (extended reality) system, the method comprising:determining a predicted pose of an XR device at a future point in time, based on data provided on the pose and motion of the XR device by one or more sensors of the XR system;preparing at least part of an image based on a notional pose of the XR device determined at least in part on the predicted pose of the XR device at the future point in time;displaying the prepared at least part of an image on a display system of the XR system at the future point in time;driving an actuator assembly to move at least part of the display system so as to reduce any differences between the displayed at least part of an image, and at least part of a target image that would have been displayed if the image preparing had been based on the pose of the XR device at the future point in time.

2. A method according to claim 1, wherein driving the actuator assembly comprises powering an SMA (shape memory alloy) element configured, when powered, to apply an actuating force to the at least part of the display system capable of moving the at least part of the display system relative to a support structure of the display system.

3. A method according to claim 1 or 2, wherein the notional pose of the XR device is determined based on: (i) the predicted pose of the XR device at the future point in time, and (ii) a determined position or a predicted position of the actuator assembly within a range of movement.

4. A method according to claim 3, wherein, if the determined or predicted position of the actuator assembly corresponds to a position at an end of the range of movement, the notional pose is determined as a pose that is expected to cause the actuator assembly to be moved towards the centre of the range of movement when the actuator assembly is driven to reduce any differences between the displayed at least part of an image, and the at least part of a target image.

5. A method according to any of claims 3 or 4, wherein the notional pose of the XR device is determined further based on: the velocity and / or acceleration capabilities of the actuator assembly.

6. A method according to claim 1 or 2, wherein the image preparing comprises rendering at least part of an image based on a predicted pose of the XR device at the future point in time determined pre rendering.P632-GB01 187. A method according to any of claims 1 to 5, wherein the image preparing comprises rendering at least part of an image based on an adjusted predicted pose of the XR device determined pre rendering, wherein the adjusted predicted pose is determined based on: (i) a predicted pose of the XR device at the future point in time determined pre rendering, and (ii) a determined position or a predicted position of the actuator assembly determined pre rendering.

8. A method according to claim 6 or 7, wherein the image preparing comprises late-warping the rendered at least part of an image based on a pose of the XR device determined post rendering.

9. A method according to claim 6 or 7, wherein the image preparing comprises late-warping the rendered at least part of an image based on an adjusted determined pose of the XR device, wherein the adjusted determined pose is determined based on: (i) a pose of the XR device determined post rendering, and (ii) a determined position or a predicted position of the actuator assembly determined post rendering.

10. A method according to claim 6 or 7, wherein the image preparing comprises late-warping the rendered at least part of an image based on a predicted pose of the XR device at the future point in time determined post rendering.

11. A method according to claim 6 or 7, wherein the image preparing comprises late-warping the rendered at least part of an image based on an adjusted predicted pose of the XR device determined post rendering, wherein the adjusted predicted pose is based on: (i) a predicted pose of the XR device at the future point in time determined post rendering, and (ii) a determined position or a predicted position of the actuator assembly determined post rendering.

12. A method according to any preceding claim, wherein the at least part of the display system is moved to compensate for any differences between the notional pose and the pose of the XR device at the future point in time.

13. A method according to any preceding claim, wherein the prepared at least part of an image is a line of an image; wherein the display system is a line sequential display system; and wherein the method comprises carrying out the steps of predicting a future pose, image preparing, image displaying, and driving the actuator assembly, for each line of the line sequential display system.

14. A method according to any of claims 1 to 12, wherein the prepared at least part of an image is a sub-frame for a colour component of an image; wherein the display system is a colour sequential displayP632-GB01 19system; and wherein the method comprises carrying out the steps of predicting a future pose, image preparing, image displaying, and driving the actuator assembly, for each colour component of the colour sequential display system.

15. A method according to any of claims 1 to 12, wherein the prepared at least part of an image is a complete image, and wherein the display system is a simultaneous display system.

16. A method according to claim 14 or 15, comprising moving the at least part of the display system so as to compensate for any motion of the XR device during the image displaying.

17. A method according to any preceding claim, wherein the method is a method of reducing the perceived motion-to-photon latency of an XR system to 20 ms or less, to 10 ms or less, or to 5 ms or less.

18. A method according to any preceding claim, wherein the one or more sensors comprises at least one visual sensor and at least one inertial sensor; and wherein the determining of a predicted pose of the XR device at a future point in time comprises using data provided by the at least one visual sensor and the at least one inertial sensor.

19. A computer-readable medium storing instructions for causing a system to perform a method according to any of claims 1 to 18.

20. An XR (extended reality) system comprising:an XR device;one or more sensors configured to provide data on the pose and motion of the XR device; a predictor configured to determine a predicted pose of the XR device at a future point in time, based on data provided by the one or more sensors;an image processor configured to prepare at least part of an image based on a notional pose of the XR device determined at least in part based on the predicted pose of the XR device at the future point in time;a display system configured to display the prepared image at the future point in time; and an actuator assembly configured to move at least part of the display system so as to reduce the perceived motion-to-photon latency of the XR system.

21. An XR system according to claim 20, wherein the actuator assembly comprises an SMA (shape memory alloy) element configured, when powered, to apply an actuating force to the at least part of theP632-GB01 20display system capable of moving the at least part of the display system relative to a support structure of the display system.

22. An XR system according to claim 20 or 21, wherein the image processor is configured to render at least part of an image.

23. An XR system according to claim 22, wherein the image processor is configured to late-warp the rendered at least part of an image.

24. An XR system according to any of claims 20 to 23, wherein the one or more sensors comprises at least one visual sensor and at least one inertial sensor.

25. An XR system according to any of claims 20 to 24, comprising a computer-readable medium according to claim 19.

Images