Pose updating in extended reality systems

Abstract

There is provided techniques for rendering an object in an XR environment. A method is performed by an XR client device. The method comprises receiving a calculated pose of the object for a current server simulation time, and re-calculated poses of the object for a respective previous server simulation time. The method comprises predicting a pose of the object for a target display time based on the received calculated pose of the object and the received re-calculated poses of the object. The method comprises rendering, at the target display time, an XR environment comprising the object in accordance with the predicted pose of the object.

Description

[0001] POSE UPDATING IN EXTENDED REALITY SYSTEMS

[0002] TECHNICAL FIELD

[0003] Embodiments presented herein relate to methods, an extended reality server device, an extended reality client device, computer programs, and a computer program product for handling poses of an object.

[0004] BACKGROUND

[0005] In general terms, Extended Reality (XR) encompasses a range of immersive technologies, such as Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR). These technologies share a common goal of blending the physical world with a virtual world, thereby creating new environments and experiences that allow users to interact with digital content in immersive ways. One challenge across all XR systems is the accurate and timely detection of the real-world environment, which is needed for ensuring that virtual elements are correctly aligned and responsive to the user's physical surroundings.

[0006] In XR systems, detecting and understanding the real environment often involves determining the pose (i.e., position and orientation) of physical objects. Whether it is an AR application where virtual objects interact with the physical world, a VR system that needs to respond to the movements of real-world controllers or sensors, or an MR experience that blends both real and virtual objects in a shared space, knowing the pose of the physical object is critical for delivering an immersive and interactive experience.

[0007] For instance, in VR systems, user interactions are typically mediated through hand controllers, gloves, or body sensors that track the user's movements in real time. The system must detect the position and orientation of these input devices accurately to ensure that the virtual environment responds to the user's actions as expected. Similarly, in MR environments, real and virtual objects co-exist and interact in a shared space, making it essential for the system to understand the position of physical objects and the user in relation to the virtual content. This enables scenarios where, for example, a virtual object may appear to bounce off a physical object in terms of a table, or a digital character that might navigate around real-world obstacles. However, as in AR, detecting the real environment in XR systems is not instantaneous. Capturing data from sensors, processing the information to recognize objects, and determining their pose takes time. This delay can pose problems; when the detected pose is used to update the virtual environment, the pose is often already outdated. In fast-paced or highly interactive XR experiences, this delay can result in a visible and disruptive misalignment between the virtual objects and the physical objects. For example, in a VR game where a player uses a hand controller to manipulate virtual objects, a lag in detecting the controller's movement can cause the virtual object to appear delayed or out of sync, breaking the sense of immersion.

[0008] In MR, where physical objects and virtual objects share the same space and are expected to interact seamlessly, the issue of outdated pose information is even more pronounced. The virtual objects need to be continuously adjusted based on real-time data from the physical world, and any delay in updating the system with the latest pose information can lead to inconsistencies between the virtual objects and the physical objects. This can disrupt the illusion of interaction, making virtual objects appear unresponsive or misaligned with their real-world counterparts.

[0009] To mitigate these issues, XR systems often need to predict the pose of physical objects, projecting where the physical object is likely to be at the time the virtual environment is updated. Such a predictive approach can thus be applied for addressing the lag caused by the delay in environment detection and ensuring that virtual objects remain synchronized with real-world actions. By predicting future poses, the system can attempt to minimize the effect of detection delays, allowing virtual objects to respond more accurately and in real time to movements and interactions in the physical world.

[0010] Thus, whilst XR technologies offer unprecedented opportunities for blending the physical world with a virtual world, there are challenges of environment detection delays and outdated pose information across XR systems.

[0011] Hence, there is still a need for improved synchronization of pose information across XR systems. SUMMARY

[0012] An object of embodiments herein is to address the above issues in an effort to improve the synchronization of pose information across XR systems.

[0013] A particular object is to address the challenges of environment detection delays and the use of outdated pose information.

[0014] According to a first aspect there is presented an XR server device for sending poses of a second object to an XR client device. The XR server device comprises processing circuitry. The processing circuitry is configured to cause the XR server device to predict a pose of a first object for a current server simulation time based upon one or more received poses of the first object. The processing circuitry is configured to cause the XR server device to calculate a pose of a second object for the current server simulation time based on the predicted pose of first object for the current server simulation time. The processing circuitry is configured to cause the XR server device to re-predict at least one previously predicted pose of the first object for respective previous server simulation times at least based upon said one or more received poses of the first object and an update to the pose of the first object. The processing circuitry is configured to cause the XR server device to re-calculate at least one previously calculated pose of the second object for each re-predicted pose of the first object for a respective previous server simulation time and based on the at least one re-predicted pose of the first object. The processing circuitry is configured to cause the XR server device to send, to the XR client device, the calculated pose of the second object for the current server simulation time, and the at least one recalculated pose of the second object.

[0015] According to a second aspect there is presented a method for sending poses of a second object to an XR client device. The method is performed by an XR server device. The method comprises predicting a pose of a first object for a current server simulation time based upon one or more received poses of the first object. The method comprises calculating a pose of a second object for the current server simulation time based on the predicted pose of first object for the current server simulation time. The method comprises re-predicting at least one previously predicted pose of the first object for respective previous server simulation times at least based upon said one or more received poses of the first object and an update to the pose of the first object. The method comprises re-calculating at least one previously calculated pose of the second object for each re-predicted pose of the first object for a respective previous server simulation time and based on the at least one re-predicted pose of the first object. The method comprises sending, to the XR client device, the calculated pose of the second object for the current server simulation time, and the at least one re-calculated pose of the second object.

[0016] According to a third aspect there is presented a computer program for sending poses of a second object to an XR client device. The computer program comprises computer code which, when run on processing circuitry of an XR server device, causes the XR server device to perform actions. One action comprises the XR server device to predict a pose of a first object for a current server simulation time based upon one or more received poses of the first object. One action comprises the XR server device to calculate a pose of a second object for the current server simulation time based on the predicted pose of first object for the current server simulation time. One action comprises the XR server device to re-predict at least one previously predicted pose of the first object for respective previous server simulation times at least based upon said one or more received poses of the first object and an update to the pose of the first object. One action comprises the XR server device to re-calculate at least one previously calculated pose of the second object for each re-predicted pose of the first object for a respective previous server simulation time and based on the at least one re-predicted pose of the first object. One action comprises the XR server device to send, to the XR client device, the calculated pose of the second object for the current server simulation time, and the at least one re-calculated pose of the second object.

[0017] According to a fourth aspect there is presented an XR client device for rendering a second object. The XR client device comprises processing circuitry. The processing circuitry is configured to cause the XR client device to receive, from an XR server device, a calculated pose of the second object for a current server simulation time, and at least one re-calculated pose of the second object for a respective previous server simulation time. The processing circuitry is configured to cause the XR client device to predict a pose of the second object for a target display time based on the received calculated pose of the second object and the at least one received recalculated poses of the second object. The processing circuitry is configured to cause the XR client device to render, at the target display time, an XR environment comprising the second object in accordance with the predicted pose of the second object.

[0018] According to a fifth aspect there is presented a method for rendering a second object. The method is performed by an XR client device. The method comprises receiving, from an XR server device, a calculated pose of the second object for a current server simulation time, and at least one re-calculated pose of the second object for a respective previous server simulation time. The method comprises predicting a pose of the second object for a target display time based on the received calculated pose of the second object and the at least one received re-calculated poses of the second object. The method comprises rendering, at the target display time, an XR environment comprising the second object in accordance with the predicted pose of the second object.

[0019] According to a sixth aspect there is presented a computer program for rendering a second object. The computer program comprises computer code which, when run on processing circuitry of an XR client device, causes the XR client device to perform actions. One action comprises the XR client device to receive, from an XR server device, a calculated pose of the second object for a current server simulation time, and at least one re-calculated pose of the second object for a respective previous server simulation time. One action comprises the XR client device to predict a pose of the second object for a target display time based on the received calculated pose of the second object and the at least one received re-calculated poses of the second object. One action comprises the XR client device to render, at the target display time, an XR environment comprising the second object in accordance with the predicted pose of the second object.

[0020] According to a seventh aspect there is presented a computer program product comprising a computer program according to at least one of the third aspect and the sixth aspect and a computer readable storage medium on which the computer program is stored. The computer readable storage medium could be a non-transitory computer readable storage medium.

[0021] According to an eight aspect there is presented an XR system that comprises an XR server device according to the first aspect and at least one XR client device according to the fourth aspect. Advantageously, these aspects improve the synchronization of pose information between the XR server device and the XR client device, and thus across the XR system.

[0022] Advantageously, these aspects enable the calculated poses sent from the XR server device to the XR client device to be more accurate. This is because the re-calculated poses are based on more recent received poses to be used for the predictions of the pose of the first object and the (re-)calculations of the poses of the second object. Further, some predicted poses of the first object can be replaced by actually measured poses.

[0023] As the poses sent from the XR server device to the XR client device are more accurate, also the output of the prediction at the XR client device becomes more accurate. In turn, this improves the user experience. Also, because the poses sent from the XR server device to the XR client device are more accurate, both the latency and the computational complexity can be reduced at the XR client device.

[0024] Other objectives, features and advantages of the enclosed embodiments will be apparent from the following detailed disclosure, from the attached dependent claims as well as from the drawings.

[0025] Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All references to "a / an / the element, apparatus, component, means, module, step, etc." are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, module, step, etc., unless explicitly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated.

[0026] BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The inventive concept is now described, by way of example, with reference to the accompanying drawings, in which:

[0028] Fig. 1 is a schematic illustration of an XR system according to embodiments;

[0029] Fig. 2 is a schematic illustration of an XR environment according to embodiments; Fig. 3 is a block diagram of an XR system according to embodiments;

[0030] Fig. 4 is an illustration of pose calculations for a virtual object according to an example;

[0031] Fig. 5 is an illustration of pose calculations for a virtual object according to embodiments;

[0032] Figs. 6, 7, 8, 9, io, and 11 are flowcharts of methods according to embodiments;

[0033] Fig. 12 is a schematic diagram showing structural units of an XR server device according to an embodiment;

[0034] Fig. 13 is a schematic diagram showing functional modules of an XR client device according to an embodiment; and

[0035] Fig. 14 shows one example of a computer program product comprising computer readable means according to an embodiment.

[0036] DETAILED DESCRIPTION

[0037] The inventive concept will now be described more fully hereinafter with reference to the accompanying drawings, in which certain embodiments of the inventive concept are shown. This inventive concept may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the inventive concept to those skilled in the art. Like numbers refer to like elements throughout the description. Any step or feature illustrated by dashed lines should be regarded as optional.

[0038] Fig. 1 schematically illustrates an XR system 100. The XR system 100 comprises an XR server device 110 and two XR client devices 120. The XR client devices 120 and the XR server device 110 are configured to communicate with each other over communication links 130. Communication between the XR server device 110 and the XR client devices 120 is typically established via a network connection, such as Wi-Fi, cellular connection, or Ethernet. The XR server device 110 processes data, including pose information, virtual environment updates, and other computational tasks, then transmits this data to the XR client devices 120. The XR client devices 120, in turn, receive and process the transmitted data to render a virtual scene and adjust object placement in real time. Additionally, the XR client devices 120 send feedback data, such as user input or sensor readings, back to the XR server device 110, enabling continuous interaction and synchronization between real and virtual environments. In the illustrative example of Fig. 1, each XR client device 120 is an eyewear, such as a head-mounted display (HMD) device. However, there are also other types of XR client devices, such as smartphones, tablets, projection-based systems, smart glasses, immersive domes, body tracking suits, etc. However, in any case it is assumed that the XR client device 120 is capable of rendering a visual representation of an XR environment.

[0039] Fig. 2 schematically illustrates an XR environment 200. In the illustrative examples of Fig. 2, a user 210 is wearing an XR client device 220 in terms of smart glasses. The XR environment 200 comprises one physical object 230, namely a car, and one virtual object 240, namely a flag. The virtual object 240 has a pose given by coordinates in a coordinate system (x, y, z) and is assumed to follow movements of physical object 230. When the XR environment 200 is viewed through the smart glasses the flag should appear to be placed at the top of the car, as the car moves around. The movement of the physical object 230 can be registered by sensors 250a, 250b (such as cameras, LiDAR sensors, infrared sensors, time-of-flight (ToF) sensors, structured light sensors, etc.).

[0040] As disclosed above, there is a need for improved synchronization of pose information across XR systems 100.

[0041] As an illustrative example, consider the block diagram of the XR system 300 in Fig. 3 comprising a sensor 310, an XR server device 320, and an XR client device 330. As in Fig. 2, the sensor 310 is configured to detect changes in the physical part of the XR environment (e.g., movements of the physical object 230 and the XR client device 220) and estimate the corresponding poses. The time the change was detected in the physical part of the XR environment is denoted TC. The XR server device 320 comprises a physical object pose predictor 322 configured to predict the poses of the physical objects from the time TC for a simulation time TS of a game engine 324. The simulation time TS may be an approximation of the actual display time TD at which the virtual object is to be rendered by a display 336 at the XR client device 336. However, this exact display time TD is not known to the XR server device 320. The predicted poses are sent from the physical object pose predictor 322 to the game engine 324. The game engine 324 calculates the pose of the virtual object. The pose of the virtual object is then sent from the XR server device 320 to the XR client device 330. The XR client device comprises a virtual object pose predictor 332 configured to update, or correct, the pose of the virtual object (based on the history of previously received virtual object poses) to the actual display time TD, and a game engine 334 in the XR client device 330 will use this (“double-predicted”) virtual object pose either for a local rendered virtual object, or for the remote rendered image of the virtual object. This object pose correction at the XR client device 330 is a stand-alone prediction, based on previously received poses of the virtual object. The updated pose of the virtual object is thus provided to the game engine 334 in the XR client device where parameters (color, texture, etc.) for rendering the virtual object are calculated. The virtual object is then rendered at the display 336 of the XR client device. The time the virtual object is rendered is referred to as the display time TD.

[0042] In the above described scenario, the XR client device thus operates on already predicted input data (since the calculated pose of the virtual object is based on the predicted pose of the real object). This is further explained with reference to Fig. 4. In Fig. 4 is illustrated an example of pose predictions for a physical object made at the XR server device and pose predictions for a virtual object made at the XR client device, respectively. Operations at the XR server device side will be disclosed next. At step i) at the XR server device side, based on observations of the real poses Ri, R2, the XR server device predicts the pose of the real object for time T4 and also calculates the pose of the virtual object for time T4. This prediction and calculation is referred to as P1-1. Likewise, at step ii) at the XR server device side, based on observations of the real poses R2, R3, the XR server device predicts the pose of the real object for time T5 and also calculates the pose of the virtual object for time T5. This prediction and calculation is referred to as P1-2. Finally, at step iii) at the XR server device side, based on observations of the real poses R3, R4, the XR server device predicts the pose of the real object for time T6 and also calculates the pose of the virtual object for time T6. This prediction and calculation is referred to as P1-3. The real poses Ri:R4 are for illustrative purposes assumed to be of a physical object (like the car in Fig. 2). The calculations P1-1, P1-2, P1-3 are sent from the XR server device to the XR client device. Operations at the XR client device side will be disclosed next. At step i) at the XR client device side, based on the calculated poses P1-1, P1-2 of the virtual object for times T4 and T5, respectively, the XR client device predicts the pose of the virtual object for time T7. This prediction is referred to as P2- 1. Likewise, at step ii) at the XR client device side, based on the calculated poses P1-2, P-3 of the virtual object, the XR client device predicts the pose of the virtual object at time T8. This prediction is referred to as P2-2. Finally, at step iii) at the XR client device side, based on the calculated poses P1-3, P1-4 of the virtual object, the XR client device predicts the pose of the virtual object at time T9. This prediction is referred to as P2-3. Hence, P2-1 is predicted based on calculations P1-1 and P1-2, and so on. In other words, the predictions made at the XR client device are doubleprediction (i.e., a combination of real object pose prediction by the server and virtual object pose prediction by the client) which may enlarge any previous prediction errors. This setup also delays the reaction to unpredictable (e.g. user action) events. In turn, this may, for example, cause misalignment of virtual objects with respect to physical objects.

[0043] The herein disclosed embodiments are based on the XR server device being configured to send updated (hereinafter referred to as re-calculated) versions of previously sent calculated poses to the XR client device. The XR client device then uses the re-calculated versions of the poses during its own predictions. In further detail, the XR server device is to send re-calculated poses of the virtual objects not only for current target display time, but also for specific previous target times. The XR client device may then receive re-calculated poses of virtual objects for timestamps already received and use the re-calculated poses instead of the previously received poses of the virtual object for local pose predictions of the virtual object in the XR client device. In some examples the XR client device is enabled to overwrite already received values and re-run prediction with the new values. Therefore, in some examples, the XR server device sends not only re-calculated poses of a virtual object but also the timestamp the re-calculations correspond to. This requires the XR server device to be configured to recalculate the poses for N>1 target timestamps per calculated pose. The value of N may depend on how many previous poses the XR client device uses to re-predict the pose of the real object. For example, the XR server device may send the calculated, and re-calculated, poses of the virtual object at a rate corresponding to the frame rate (e.g., 60 times per second) of the rendering at the XR client device. The target time is the simulation time (TS), which could be close to the display time (TD) for the given frame.

[0044] Reference is here made to Fig. 5 which illustrates an example of pose predictions for a real object and pose calculations for a virtual object made at the XR server device and pose predictions for the virtual object made at the XR client device in accordance with embodiments as will be disclosed hereinafter. Operations at the XR server device side are illustrated in Fig. 5(a) and will be disclosed next. At step i), based on observations of the real poses Ri, R2, the XR server device predicts the pose of the real object for time T4 and also calculates the pose of the virtual object for time T4. This prediction and this calculation are referred to as P1-1. Likewise, at step ii), based on observations of the real poses R2, R3, the XR server device predicts the pose of the real object for time T5 and also calculates the pose of the virtual object for time T5. This prediction and this calculation are referred to as P1-2. In addition, based on the real pose R3, the XR server device updates the pose P1-1. The update comprises a reprediction of the pose of the real object for time T4 and a re-calculation of the pose of the virtual object for time T4. These updated poses are denoted P1-1’. Finally, at step iii) at the XR server device side, based on observations of the real poses R3, R4, the XR server device predicts the pose of the real object for time T6 and also calculates the pose of the virtual object for time T6. This prediction and this calculation are referred to as P1-3. In addition, based on the real pose R4, the XR server device updates the poses P1-1’. The update comprises a re-prediction of the pose of the real object for time T4 and a re-calculation of the pose of the virtual object for time T4. These updated poses are denoted P1-1”. Further, based on the real pose R4, the XR server device also updates the poses P1-2. The update comprises a re-prediction of the pose of the real object for time T5 and a re-calculation of the pose of the virtual object for time T5. These updated poses are denoted P1-2’. The calculations and recalculations contained in the poses P1-1, P1-2, P1-3, P1-1’, P1-1”, P1-2’ are sent from the XR server device to the XR client device. Either these calculations and recalculations are sent as soon as having been calculated, or re-calculated, or all calculations (and re-calculations) are sent once all calculations (and re-calculations) have been completed. These two alternatives will be disclosed in more detail with reference to Fig. 7 and Fig. 8, respectively.

[0045] Operations at the XR client device side are illustrated in Fig. 5(b) and will be disclosed next. It is in this example assumed that the calculations and re-calculations contained in the poses are sent from the server device as soon as having been calculated, or re-calculated. It is further assumed that the XR client device has already received P1-1. At step i), the XR client device receives re-calculated pose P1-1’ and calculated poses P1-2 for times T4 and T5, respectively. The XR client device replaces P1-1 with P1-1’. Based on P1-1’ and P1-2, the XR client device then predicts the pose of the virtual object for time T7. This prediction is referred to as P2-1. Likewise, at step ii), the XR client device receives re-calculated poses P1-1”, P1-2’ and calculated pose P1-3 for times T4, T5, and T6, respectively. The XR client device replaces P1-1’ with P1-1” and P1-2 with P1-2’. Based on P1-1”, P1-2’ and P1-3, the XR client device then predicts the pose of the virtual object for time T8. This prediction is referred to as P2-2. Finally, at step iii), the XR client device receives re-calculated poses P1-2”, P1-3’ and calculated pose P1-4 for times T5, T6, and T7, respectively. The XR client device replaces P1-2’ with P1-2” and P1-3 with P1-3’. Based on P1-3”, P1-3’ and P1-4, the XR client device then predicts the pose of the virtual object for time T9. This prediction is referred to as P2-3. Hence, whilst predictions P2-2 and P2-3 are based on three values, prediction P2-1 is based on only two values. This is due to initialization where thus P1-1 is the first prediction (where P1-1’ replaces P1-1). It is here noted that the server device and the client device may not be in perfect synchronization with each other. As an illustrative example, this might imply that the server device’s understanding of time Ti may not be exactly the same as the client device’s understanding of time Ti, and so on. However, the associated timestamps are assumed to be absolute timestamps (e.g., not just specifying “Ti”, but “2024-11- 05 12:01:06.234 UTC”, which can be correctly interpreted at the receiving end (i.e., at the client device).

[0046] Reference is now made to Fig. 6 illustrating a method for sending poses of a second object 240 to an XR client device 120, 220 as performed by the XR server device 110 according to an embodiment. S102: The XR server device 110 predicts a pose of a first object 230 for a current server simulation time based upon one or more received poses of the first object (230).

[0047] S104: The XR server device 110 calculates a pose of a second object 240 for the current server simulation time based on the predicted pose of first object 230 for the current server simulation time.

[0048] S106: The XR server device 110 re-predicts at least one previously predicted pose of the first object 230 for respective previous server simulation times at least based upon the one or more received poses of the first object 230 and an update to the pose of the first object 230. The update may be an update to at least one of the one or more received poses of the first object 230.

[0049] S108: The XR server device 110 re-calculates at least one previously calculated pose of the second object 240 for each re-predicted pose of the first object 230 for a respective previous server simulation time and based on the at least one re-predicted pose of the first object 230.

[0050] S110: The XR server device 110 sends, to the XR client device 120, 220, the calculated pose of the second object 240 for the current server simulation time, and the at least one re-calculated pose of the second object 240.

[0051] Typically, the first object 230 is a real object and the second object 240 is a virtual object, as in the example of Fig. 2. Alternatively, also the first object 230 is a virtual object. Yet alternatively, the first object 230 and the second object 240 is one and the same object. In this case the re-calculation step S108 can be omitted since it would involve the same calculations as in step S106.

[0052] Embodiments relating to further details of sending poses of a second object 240 to an XR client device 120, 220 as performed by the XR server device 110 will now be disclosed.

[0053] As disclosed above, in some examples, not only the poses are sent but also the timestamps the poses correspond to. Hence, in some embodiments, the calculated pose and the at least one re-calculated pose are sent together with respective timestamps.

[0054] In some aspects, the poses are calculated for the current simulation timestamp and recalculated for the previous N timestamps. That is, in some embodiments, there are JV>I re-calculated poses.

[0055] As disclosed above, the value of N may depend on how many previous poses the XR client device uses to predict the pose at the time TD. That is, in some embodiments, the value of N depends on how many re-predicted pose of the first object 230 are used by the XR client device 120, 220 when re-calculating the at least one previously calculated pose of the second object 240.

[0056] In some aspects, the XR server device 110 re-predicts previously already predicted object poses based on updated environment information (e.g., as received from one or more sensor 250a, 250b). Hence, in some embodiments, the one or more received poses of the first object 230 are based on information in the XR environment 200 as updated for the simulation time.

[0057] As in the example of Fig. 5(a), some of the predicted pose values can be replaced by actually measured values, as time passes. Hence, in some embodiments, at least one of the one or more received poses of the first object 230 is an actual pose of the first object 230 for a previous server simulation times.

[0058] Optionally, an index number is sent together with each re-calculated pose value. That is, in some embodiments, the at least one re-calculated pose is sent together with a respective index number. The index number for a given re-calculated pose reflects how many times this given re-calculated pose has been re-calculated. The index number for a given re-calculated pose is strictly monotonically increasingwith each virtual object pose re-calculation. The index number could be a server simulation timestamp, simulation iteration counter, server wall clock time value, or the like.

[0059] How the XR client device 120, 220 may use the index number will be disclosed below.

[0060] There could be different ways for the XR server device 110 to send the re-calculated poses to the XR client device 120, 220. In a first embodiment, each of the at least one re-calculated pose is sent to the XR client device 120, 220 as soon as having been calculated. This embodiment is illustrated in the flowchart of Fig. 7, as will be described next. The current simulation timestamp is denoted “n” and the method is repeated for the previous N timestamps. The method is performed by the XR server device 110.

[0061] S201: A time index i is reset to zero; i = o.

[0062] S202: The pose (Pr) of a physical object is predicted for simulation time Ts[n-i].

[0063] S203: The pose (Pv) of the virtual object is calculated based on the predicted pose (Pr) of the physical object.

[0064] S204: The calculated pose (Pv) of the virtual object and the simulation time value Ts[n-1] are sent to the XR client device.

[0065] S205: It is checked whether the current value of i is smaller than N, i.e., if i < N or not. If yes, then step S206 is entered. If no, then the procedure is ended.

[0066] S206: The value of i is incremented; i = i+1.

[0067] In a second embodiment, all of the at least one re-calculated pose prediction is sent to the XR client device 120, 220 once all of the at least one re-calculated pose prediction have been calculated. This embodiment is illustrated in the flowchart of Fig. 8, as will be described next. The current simulation timestamp is denoted “n” and the method is repeated for the previous N timestamps. The method is performed by the XR server device 110.

[0068] S301: A list of pose values and time values is cleared; [Pv, Ts] ={}. A time index i is reset to zero; i = o.

[0069] S302: The pose (Pr) of a physical object is predicted for simulation time Ts[n-i].

[0070] S303: The value of the pose and the simulation time value are added to the list.

[0071] S304: It is checked whether the current value of i is smaller than N, i.e., if i < N or not. If yes, then step S305 is entered. If no, then step S307 is entered.

[0072] S305: The value of i is incremented; i = i+1. Step S302 is then entered again. S306: The pose (Pv) of the virtual object is calculated based on all the predicted poses of the physical object in the list, with one value of the pose of the virtual object being calculated for each simulation time value.

[0073] S307: The list of calculated poses of the virtual object and the corresponding simulation time values are sent to the XR client device.

[0074] Two example scenarios will be described next to illustrate how multiple pose calculations could be implemented.

[0075] In a first example scenario, there is one virtual object (e.g. a virtual flag) placed at the roof of a physical object (e.g. a detected car) in the XR environment. This represents a simple relation between the pose of the physical object and the pose of the virtual object, which can be described e.g. by a simple mathematical equation. For example, the pose, Pv(t), of the virtual object at time t can be described by a function of the pose(s) or the physical object(s); Pv(t)=f(PRi(t), PRa(t), ...)).

[0076] In a second example scenario, the XR environment has more complex content, e.g. multiple (physical and / or) virtual objects. Although the pose of the physical object has impact on the pose of the virtual object, it is here assumed that a simple equation cannot describe the relation between the pose of the virtual object and the pose of the physical object. One reason for this could be that several (physical and / or) virtual objects can have actions and relations with respect to each other. Thus, the poses of all physical objects have to be predicted to the simulation time, and then the game engine has to run predictions to get the pose(s) of the virtual object(s).

[0077] Three example options for implementing the calculations of the poses of the virtual object for multiple target times as in the second example scenario will be disclosed next.

[0078] In a first example option, there are corresponding server instances to different simulation time Ts, which is Ts[a, a-1, ..., a-IV]. The first server instance with the actual simulation time (TS [a] ) is really used, and it represents the game state. The other server instances are used only as the prediction helper, to create re-calculated poses for their representative times. In a second example option, there is a single server instance, but there are multiple XR environments, all of them very similar except of the simulation time Ts, which is Ts[a, a-i, ..., a-IV]. All XR environments receive the most recent pose from the object detection and also use it. The first XR environment with the actual simulation time (Ts[a]) is really used, and it represents the game state. The other XR environments are used only as the prediction helper, to create updated predicted poses for their time.

[0079] In a third example option, there is a single server with a single XR environment, but multiple instances of all objects, corresponding to different simulation time Ts, which is Ts[a, a-i, ..., a-IV]. The first instance with the actual simulation time (Ts[a]) is really used, and it represents the game state. The other server instances are used only as the prediction helper, to create re-calculated poses for their representative times.

[0080] Reference is now made to Fig. 9 illustrating a method for updating pose of a virtual object 240 as performed by the XR client device 120, 220 according to an embodiment.

[0081] S402: The XR client device 120, 220 receives, from the XR server device (110, 1200), a calculated pose of the second object (240) for a current server simulation time, and at least one re-calculated pose of the second object (240) for a respective previous server simulation time.

[0082] S404: The XR client device 120, 220 predict a pose of the second object 240 for a target display time based on the received calculated pose of the second object 240 and the at least one received re-calculated poses of the second object 240.

[0083] S406: The XR client device 120, 220 renders, at the target display time, an XR environment 200 comprising the second object 240 in accordance with the predicted pose of the second object 240.

[0084] Embodiments relating to further details of rendering the second object 240 as performed by the XR client device 120, 220 will now be disclosed.

[0085] As disclosed above, in some examples, not only the poses are sent but also the timestamps the poses correspond to. Hence, in some embodiments, the calculated pose and the at least one re-calculated pose are received together with respective timestamps.

[0086] In some aspects, the poses are calculated for the current simulation timestamp and recalculated for the previous N timestamps. That is, in some embodiments, there are JV>I re-calculated poses.

[0087] In some aspects, the XR client device 120, 220 receives re-calculated poses of the second object 240 for timestamps already received and uses the re-calculated poses instead of the previously received poses for local pose prediction. In particular, in some embodiments, a corresponding pose is replaced by the at least one re-calculated pose in accordance with the timestamps when predicting the pose of the second object 240.

[0088] As disclosed above, optionally, an index number is sent together with each recalculated pose value. Thus, in some embodiments, the at least one re-calculated pose is received together with respective index numbers. The index number for a given recalculated pose reflects how many times this given re-calculated pose has been recalculated. In particular, in some embodiments, a given previous pose of the second object 240 is replaced by a given re-calculated pose upon the XR client device 120, 22ohaving verified that the index number of the given re-calculated pose indicates that the given re-calculated pose is newer than the given previous pose of the second object 240. In this way, it can be ensured that a (re-)calculated pose prediction is only overwritten by another re-calculated pose that is fresher (i.e., newer).

[0089] As follows from the above, the XR client device 120, 220 is enabled to receive recalculated poses of a second object for timestamps already received and use the recalculated poses instead of the previous ones for local pose predictions. This requires the XR client device 120, 220 to be able to overwrite already received (re-)calculated values and re-run prediction with new re-calculated values, in Fig. 5(b). However, as there could be different ways for the XR server device 110 to send the re-calculated poses to the XR client device 120, 220, there could also be different ways for the XR client device 120, 220 to receive the re-calculated poses from the XR server device 110. In a first embodiment, corresponding to the embodiment in Fig. 7, each of the at least one re-calculated poses is received by the XR client device 120, 220 as soon as having been calculated at the XR server device 110. This embodiment is illustrated in the flowchart of Fig. 10 and will be described next. The method is performed by the XR client device 120, 220.

[0090] S501: The XR client device 120, 220 receives the re-calculated pose (Pv) of the virtual object for the simulation time value Ts.

[0091] S502: The received simulation time value Ts may not correspond exactly to an internal simulation time value, denoted Tstored, in the XR client device 120, 220. The XR client device 120, 220 therefore finds the internal simulation time value that is closest to the received simulation time value.

[0092] S503: It is checked whether the absolute difference between the internal simulation time value and the received simulation time value is larger than some threshold value or not. If yes, step S504 is entered. If no, step S505 is entered.

[0093] S504: The received re-calculated pose and the received simulation time value are added to a list.

[0094] S505: The received re-calculated pose and the received simulation time value overwrite the corresponding pair of ( recalculated pose and simulation time value for the virtual object.

[0095] S506: It is checked whether some additional re-prediction of the pose is needed. If yes, step S507 is entered. If no, step S501 is entered again for the next simulation time value.

[0096] S507: The pose of the virtual object is re-predicted for the actual display time.

[0097] In a second embodiment, corresponding to the embodiment in Fig. 8, all of the at least one re-calculated pose is received by the XR client device 120, 220 once all of the at least one re-calculated pose have been calculated at the XR server device 110. This embodiment is illustrated in the flowchart of Fig. 11, as will be described next. The method is performed by the XR client device 120, 220. S601: The XR client device 120, 220 receives a list [Pv, Ts] of re-calculated poses Pv of the virtual object with corresponding simulation time values Ts.

[0098] S602: The XR client device 120, 220 deletes all previously received poses of the virtual object.

[0099] S603: The XR client device 120, 220 stores the received list [Pv, Ts] for the virtual object.

[0100] S604: It is checked whether some additional re-prediction of the pose is needed. If yes, step S605 is entered. If no, step S601 is entered again for the next simulation time value.

[0101] S605: The pose of the virtual object is re-predicted for the actual display time.

[0102] For both the first embodiment and the second embodiment, if an index (as specified above) is received and stored together with the [Ts, Pv] pair, the previously stored index is overwritten only if the newly received index is greater than the stored one.

[0103] Fig. 12 schematically illustrates, in terms of a number of structural units, the components of an XR server device 1200 according to an embodiment. Processing circuitry 1210 is provided using any combination of one or more of a suitable central processing unit (CPU), multiprocessor, microcontroller, digital signal processor (DSP), etc., capable of executing software instructions stored in a computer program product 1410a (as in Fig. 14), e.g. in the form of a storage medium 1230. The processing circuitry 1210 may further be provided as at least one application specific integrated circuit (ASIC), or field programmable gate array (FPGA).

[0104] Particularly, the processing circuitry 1210 is configured to cause the XR server device 1200 to perform a set of operations, or steps, as disclosed above. For example, the storage medium 1230 may store the set of operations, and the processing circuitry 1210 may be configured to retrieve the set of operations from the storage medium 1230 to cause the XR server device 1200 to perform the set of operations. The set of operations may be provided as a set of executable instructions. Thus the processing circuitry 1210 is thereby arranged to execute methods as herein disclosed. The storage medium 1230 may also comprise persistent storage, which, for example, can be any single one or combination of magnetic memory, optical memory, solid state memory or even remotely mounted memory.

[0105] The XR server device 1200 may further comprise a communications (comm.) interface 1220 for communications with XR client devices, sensors, etc. As such the communications interface 1220 may comprise one or more transmitters and receivers, comprising analogue and digital components.

[0106] The processing circuitry 1210 controls the general operation of the XR server device 1200 e.g. by sending data and control signals to the communications interface 1220 and the storage medium 1230, by receiving data and reports from the communications interface 1220, and by retrieving data and instructions from the storage medium 1230. Other components, as well as the related functionality, of the XR server device 1200 are omitted in order not to obscure the concepts presented herein.

[0107] Fig. 13 schematically illustrates, in terms of a number of structural units, the components of an XR client device 1300 according to an embodiment. Processing circuitry 1310 is provided using any combination of one or more of a suitable central processing unit (CPU), multiprocessor, microcontroller, digital signal processor (DSP), etc., capable of executing software instructions stored in a computer program product 1410b (as in Fig. 14), e.g. in the form of a storage medium 1330. The processing circuitry 1310 may further be provided as at least one application specific integrated circuit (ASIC), or field programmable gate array (FPGA).

[0108] Particularly, the processing circuitry 1310 is configured to cause the XR client device 1300 to perform a set of operations, or steps, as disclosed above. For example, the storage medium 1330 may store the set of operations, and the processing circuitry 1310 may be configured to retrieve the set of operations from the storage medium 1330 to cause the XR client device 1300 to perform the set of operations. The set of operations may be provided as a set of executable instructions. Thus the processing circuitry 1310 is thereby arranged to execute methods as herein disclosed. The storage medium 1330 may also comprise persistent storage, which, for example, can be any single one or combination of magnetic memory, optical memory, solid state memory or even remotely mounted memory.

[0109] The XR client device 1300 may further comprise a communications interface 1320 for communications with XR server devices, other XR client devices, etc. As such the communications interface 1320 may comprise one or more transmitters and receivers, comprising analogue and digital components.

[0110] The processing circuitry 1310 controls the general operation of the XR client device 1300 e.g. by sending data and control signals to the communications interface 1320 and the storage medium 1330, by receiving data and reports from the communications interface 1320, and by retrieving data and instructions from the storage medium 1330. Other components, as well as the related functionality, of the XR client device 1300 are omitted in order not to obscure the concepts presented herein.

[0111] Fig. 14 shows one example of a computer program product 1410a, 1410b comprising computer readable means 1430. On this computer readable means 1430, a computer program 1420a can be stored, which computer program 1420a can cause the processing circuitry 1210 and thereto operatively coupled entities and devices, such as the communications interface 1220 and the storage medium 1230, to execute methods according to embodiments described herein. The computer program 1420a and / or computer program product 1410a may thus provide means for performing any steps of the XR server device 110, 1200 as herein disclosed. On this computer readable means 1430, a computer program 1420b can be stored, which computer program 1420b can cause the processing circuitry 1310 and thereto operatively coupled entities and devices, such as the communications interface 1320 and the storage medium 1330, to execute methods according to embodiments described herein. The computer program 1420b and / or computer program product 1410b may thus provide means for performing any steps of the XR client device 120, 220, 1300 as herein disclosed.

[0112] In the example of Fig. 14, the computer program product 1410a, 1410b is illustrated as an optical disc, such as a CD (compact disc) or a DVD (digital versatile disc) or a Blu-Ray disc. The computer program product 1410a, 1410b could also be embodied as a memory, such as a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or an electrically erasable programmable read-only memory (EEPROM) and more particularly as a non-volatile storage medium of a device in an external memory such as a USB (Universal Serial Bus) memory or a Flash memory, such as a compact Flash memory. Thus, while the computer program 1420a, 1420b is here schematically shown as a track on the depicted optical disk, the computer program 1420a, 1420b can be stored in any way which is suitable for the computer program product 1410a, 1410b.

[0113] The inventive concept has mainly been described above with reference to a few embodiments. However, as is readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the inventive concept, as defined by the appended patent claims.

Claims

CLAIMS1. An extended reality, XR, server device (110, 1200) for sending poses of a second object (240) to an XR client device (120, 220, 1300), wherein the XR server device (110, 1200) comprises processing circuitry (1210) configured to cause the XR server device (110, 1200) to: predict a pose of a first object (230) for a current server simulation time based upon one or more received poses of the first object (230); calculate a pose of a second object (240) for the current server simulation time based on the predicted pose of first object (230) for the current server simulation time; re-predict at least one previously predicted pose of the first object (230) for respective previous server simulation times at least based upon said one or more received poses of the first object (230) and an update to the pose of the first object (230); re-calculate at least one previously calculated pose of the second object (240) for each re-predicted pose of the first object (230) for a respective previous server simulation time and based on the at least one re-predicted pose of the first object (230); and send, to the XR client device (120, 220, 1300), the calculated pose of the second object (240) for the current server simulation time, and the at least one re-calculated pose of the second object (240).

2. The XR server device (110, 1200) according to claim 1, wherein the calculated pose and the at least one re-calculated pose are sent together with respective timestamps.

3. The XR server device (110, 1200) according to claim 1, wherein there are V>1 recalculated poses.

4. The XR server device (110, 1200) according claim 3, wherein a value of N depends on how many re-predicted pose of the first object (230) are used by the XRclient device (120, 220, 1300) when re-calculating the at least one previously calculated pose of the second object (240).

5. The XR server device (110, 1200) according to claim 1, wherein said one or more received poses of the first object (230) are based on information in the XR environment (200) as updated for the simulation time.

6. The XR server device (110, 1200) according to claim 1, wherein at least one of the one or more received poses of the first object (230) is an actual pose of the first object (230) for a previous server simulation times.

7. The XR server device (110, 1200) according to claim 1, wherein the at least one re-calculated pose is sent together with a respective index number, wherein the index number for a given re-calculated pose reflects how many times this given recalculated pose has been re-calculated.

8. The XR server device (110, 1200) according to claim 1, wherein each of the at least one re-calculated pose is sent to the XR client device (120, 220, 1300) as soon as having been re-calculated.

9. The XR server device (110, 1200) according to claim 1, wherein all of the at least one re-calculated pose is sent to the XR client device (120, 220, 1300) once all of the at least one re-calculated pose have been re-calculated.

10. An extended reality, XR, client device (120, 220, 1300) for rendering a second object (240), wherein the XR client device (120, 220, 1300) comprises processing circuitry (1310) configured to cause the XR client device (120, 220, 1300) to: receive, from an XR server device (110, 1200), a calculated pose of the second object (240) for a current server simulation time, and at least one re-calculated pose of the second object (240) for a respective previous server simulation time; and predict a pose of the second object (240) for a target display time based on the received calculated pose of the second object (240) and the at least one received recalculated poses of the second object (240); and render, at the target display time, an XR environment (200) comprising the second object (240) in accordance with the predicted pose of the second object (240).

11. The XR client device (120, 220, 1300) according to claim 10, wherein the calculated pose and the at least one re-calculated pose are received together with respective timestamps.

12. The XR client device (120, 220, 1300) according to claim 11, wherein there areV>1 re-calculated poses.

13. The XR client device (120, 220, 1300) according to any of claims 11 or 12, wherein a corresponding pose is replaced by the at least one re-calculated pose in accordance with the timestamps when predicting the pose of the second object (240).

14. The XR client device (120, 220, 1300) according to claim 10, wherein the at least one re-calculated pose is received together with respective index numbers, wherein the index number for a given re-calculated pose reflects how many times this given re-calculated pose has been re-calculated.

15. The XR client device (120, 220, 1300) according to claim 14, wherein a given previous pose of the second object (240) is replaced by a given re-calculated pose upon the XR client device (120, 220, 1300) having verified that the index number of the given re-calculated pose indicates that the given re-calculated pose is newer than the given previous pose of the second object (240).

16. The XR client device (120, 220, 1300) according to claim 10, wherein the XR client device (120, 220, 1300) is, is part of, or is integrated with, an eyewear, such as a head-mounted display, HMD, device.

17. An extended reality, XR, system (100), the XR system (100) comprising an XR server device (110, 1200) according to any of claims 1-9 and at least one XR client device (120, 220, 1300) according to any of claims 10-16.

18. A method for sending poses of a second object (240) to an extended reality, XR, client device (120, 220, 1300), the method being performed by an XR server device (110, 1200), the method comprising: predicting (S102) a pose of a first object (230) for a current server simulation time based upon one or more received poses of the first object (230);calculating (S104) a pose of a second object (240) for the current server simulation time based on the predicted pose of first object (230) for the current server simulation time; re-predicting (S106) at least one previously predicted pose of the first object (230) for respective previous server simulation times at least based upon said one or more received poses of the first object (230) and an update to the pose of the first object (230); re-calculating (S108) at least one previously calculated pose of the second object (240) for each re-predicted pose of the first object (230) for a respective previous server simulation time and based on the at least one re-predicted pose of the first object (230); and send (S110), to the XR client device (120, 220, 1300), the calculated pose of the second object (240) for the current server simulation time, and the at least one recalculated pose of the second object (240).

19. A method for rendering a second object (240), the method being performed by an extended reality, XR, client device (120, 220, 1300), the method comprising: receiving (S402), from an XR server device (110, 1200), a calculated pose of the second object (240) for a current server simulation time, and at least one recalculated pose of the second object (240) for a respective previous server simulation time; and predicting (S404) a pose of the second object (240) for a target display time based on the received calculated pose of the second object (240) and the at least one received re-calculated poses of the second object (240); and rendering (S406), at the target display time, an XR environment (200) comprising the second object (240) in accordance with the predicted pose of the second object (240).

20. A computer program (1420a) for sending poses of a second object (240) to an extended reality, XR, client device (120, 220, 1300), the computer programcomprising computer code which, when run on processing circuitry (210) of an XR server device (110, 1200), causes the XR server device (110, 1200) to: predict (S102) a pose of a first object (230) for a current server simulation time based upon one or more received poses of the first object (230); calculate (S104) a pose of a second object (240) for the current server simulation time based on the predicted pose of first object (230) for the current server simulation time; re-predict (S106) at least one previously predicted pose of the first object (230) for respective previous server simulation times at least based upon said one or more received poses of the first object (230) and an update to the pose of the first object (230); re-calculate (S108) at least one previously calculated pose of the second object (240) for each re-predicted pose of the first object (230) for a respective previous server simulation time and based on the at least one re-predicted pose of the first object (230); and send (S110), to the XR client device (120, 220, 1300), the calculated pose of the second object (240) for the current server simulation time, and the at least one recalculated pose of the second object (240).

21. A computer program (1420b) for rendering a virtual object (240), the computer program comprising computer code which, when run on processing circuitry (310) of an extended reality, XR, client device (120, 220, 1300), causes the XR client device (120, 220, 1300) to: receive (S402), from an XR server device (110, 1200), a calculated pose of the second object (240) for a current server simulation time, and at least one recalculated pose of the second object (240) for a respective previous server simulation time; and predict (S404) a pose of the second object (240) for a target display time based on the received calculated pose of the second object (240) and the at least one received re-calculated poses of the second object (240); andrender (S406), at the target display time, an XR environment (200) comprising the second object (240) in accordance with the predicted pose of the second object (240).

22. A computer program product (1410a, 1410b) comprising a computer program (1420a, 1420b) according to at least one of claims 20 and 21, and a computer readable storage medium (1430) on which the computer program is stored.

Images