Extended reality program handling

Abstract

There is provided a computer-implemented method for handling a program for an extended reality (XR) device. The method comprises obtaining (302), by a first application programming interface (API) layer, a first request for initiation of a runtime system, wherein the first request is requested by a first XR device. The method comprises obtaining (304), by the first API layer, first information indicative of one or more characteristics of the first XR device, and obtaining (306), by the first API layer, a second request for the runtime system to execute an original program. The original program is configured to be executed to generate an output at the first XR device. The method comprises, in response to obtaining (306) the second request, modifying (308), by the first API layer, the original program based on the first information to generate a modified program, and providing (310), by the first API layer, the modified program to the runtime system for execution of the modified program.

Description

[0001] EXTENDED REALITY PROGRAM HANDLING

[0002] Technical Field

[0003] The present disclosure relates to methods for handling a program for an XR device, and entities and systems configured to operate in accordance with those methods.

[0004] Background

[0005] An extended reality (XR) device is a device which can be used to interact with extended reality. Extended reality technology is configured to combine and / or mirror the physical world with a digital world, providing users of the technology with an immersive digital experience (e.g. by being in a virtual or augmented environment). Heterogeneity in XR devices refers to the wide variety of hardware configurations, display technologies, input methods, and form factors found across different XR devices. The diversity found among XR devices presents unique challenges for developers and content creators aiming to create applications that can run seamlessly on various XR devices and / or XR platforms. Some of the key aspects of XR devices, which contribute to the aforementioned heterogeneity, include display size and resolution, display technologies, sensor types, input types, tracking systems, form factors, and ergonomics.

[0006] Addressing the heterogeneity of XR devices requires a holistic approach to application development, encompassing adaptive content rendering, input abstraction layers, device profiling, and user experience optimization. As such, in order to create applications that offer consistent, compelling experiences across a broad spectrum of XR devices, and ultimately advance the accessibility and appeal of XR technology, developers must navigate the complexities of heterogeneous XR hardware.

[0007] Some existing techniques aims at reducing the burden of XR device heterogeneity. For example, OpenXR is an open, royalty-free standard developed by the Khronos Group for creating portable, cross-platform XR applications. The goal of OpenXR is to provide a common set of application programming interfaces (APIs) that enable developers to create XR applications that can run on different hardware platforms and operating systems without requiring significant modifications. This standardization helps to reduce fragmentation in the XR ecosystem, making it easier for developers to target multiple devices and platforms with their applications.However, existing techniques, such as those associated with OpenXR, still present challenges to developers of XR technology in that the developers must manually and / or programmatically adapt each XR application to the configuration and / or capabilities of each specific XR device.

[0008] Summary

[0009] As mentioned above, there are certain challenges associated with existing techniques for handling applications for XR devices. In particular, XR devices are highly heterogeneous. That is, XR devices come in various different forms and factors which dictate their capabilities. For example, some XR devices have high-definition displays, while others have low-definition, monochromatic displays. Typically, an XR application written for the former device delivers poor experience on the latter device, and vice versa. Some current techniques attempt to address these challenges by developing a common code base, as well as an adaptation code to target each XR device individually. However, this approach results in inefficiency in development and is thus time-consuming and resource wasteful.

[0010] It is therefore an objective of the disclosure to obviate or eliminate the above-described disadvantages associated with existing techniques.

[0011] Therefore, according to an aspect of the disclosure, there is provided a first method for handling a program for an XR device. The first method is a computer-implemented method. The first method comprises obtaining, by a first application programming interface (API) layer, a first request for initiation of a runtime system. The first request is requested by a first XR device. The first method comprises obtaining, by the first API layer, first information indicative of one or more characteristics of the first XR device. The first method comprises obtaining, by the first API layer, a second request for the runtime system to execute an original program. The original program is configured to be executed to generate an output at the first XR device. The first method comprises, in response to obtaining the second request, modifying, by the first API layer, the original program based on the first information to generate a modified program. The first method comprises, in response to obtaining the second request, providing, by the first API layer, the modified program to the runtime system for execution of the modified program.

[0012] According to another aspect of the disclosure, there is provided a second method for handling a program for an XR device. The second method is a computer-implementedmethod. The second method comprises providing, by a first XR device, a first request for initiation of a runtime system. The second method comprises providing, by the first XR device, a second request for the runtime system to execute an original program. The original program is configured to be executed to generate an output at the first XR device. The second method comprises, in response to providing the second request, obtaining, by the first XR device, one or more instructions from the runtime system. The one or more instructions are generated by execution of a modified program. The modified program corresponds to a modified version of the original program. The original program is modified, by a first API layer, based on first information indicative of one or more characteristics of the first XR device. The second method comprises executing, by the first XR device, the one or more instructions.

[0013] According to another aspect of the disclosure, there is provided a method performed by a system. The method performed by the system comprises the first method referred to herein and / or the second method referred to herein.

[0014] According to another aspect of the disclosure, there is provided a first entity comprising processing circuitry configured to operate in accordance with the first method referred to herein. In some examples, the first entity may comprise at least one memory for storing instructions which, when executed by the processing circuitry, cause the first entity to operate in accordance with the first method referred to herein.

[0015] According to another aspect of the disclosure, there is provided a second entity comprising processing circuitry configured to operate in accordance with the second method referred to herein. In some examples, the second entity may comprise at least one memory for storing instructions which, when executed by the processing circuitry, cause the second entity to operate in accordance with the second method referred to herein.

[0016] According to another aspect of the disclosure, there is provided a system. The system comprises the first entity, as referred to herein, and the second entity, as referred to herein.

[0017] According to another aspect of the disclosure, there is provided a computer program comprising instructions which, when executed by processing circuitry, cause theprocessing circuitry to perform the first method referred to herein, and / or the second method referred to herein.

[0018] According to another aspect of the disclosure, there is provided a computer program product, embodied on a non-transitory machine-readable medium, comprising instructions which are executable by processing circuitry to cause the processing circuitry to perform the first method referred to herein, and / or the second method referred to herein.

[0019] Thus, in the manner described above, improved techniques for handling a program for an XR device are provided. The techniques described herein enable the automatic adaptation of an original program (e.g. XR application) to heterogenous XR devices with different capabilities (e.g. display size, display resolution, etc.). The first API layer referred to here can translate one or more components of an (e.g. generic) original program to fit the specific capabilities of a specific XR device based on one or more characteristics of the specific XR device. In this way, the techniques described herein provide for more efficient handling of programs (e.g. applications) for XR devices and result in an improved user experience for users of a wide variety of different XR devices.

[0020] Brief description of the drawings

[0021] For a better understanding of the techniques, and to show how they may be put into effect, reference will now be made, by way of example, to the accompanying drawings, in which:

[0022] Figure 1 is an illustration of an existing XR architecture;

[0023] Figure 2 is a signalling diagram illustrating an existing technique for handling applications forXR devices;

[0024] Figure 3 is a block diagram illustrating a first entity according to an embodiment;

[0025] Figure 4 is a block diagram illustrating a method performed by the first entity according to an embodiment;

[0026] Figure 5 is a block diagram illustrating a second entity according to an embodiment;Figure 6 is a block diagram illustrating a method performed by the second entity according to an embodiment;

[0027] Figure 7 is a signalling diagram illustrating an exchange of signals in a system according to an embodiment; and

[0028] Figure 8 is an illustration of a system according to an embodiment.

[0029] Detailed Description

[0030] Generally, all terms used herein are to be interpreted according to their ordinary meaning in the relevant technical field, unless a different meaning is clearly given and / or is implied from the context in which it is used. All references to a / an / the element, apparatus, component, means, step, etc. are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, step, etc., unless explicitly stated otherwise. The steps of any methods disclosed herein do not have to be performed in the exact order disclosed, unless a step is explicitly described as following or preceding another step and / or where it is implicit that a step must follow or precede another step. Any feature of any of the embodiments disclosed herein may be applied to any other embodiment, wherever appropriate. Likewise, any advantage of any of the embodiments may apply to any other embodiments, and vice versa. Other objectives, features and advantages of the enclosed embodiments will be apparent from the following description.

[0031] Some of the embodiments contemplated herein will now be described more fully with reference to the accompanying drawings. Other embodiments, however, are contained within the scope of the subject-matter disclosed herein, the disclosed subject-matter should not be construed as limited to only the embodiments set forth herein; rather, these embodiments are provided by way of example to convey the scope of the subject-matter to those skilled in the art.

[0032] The techniques described herein relate to XR devices. An XR device, as referred to herein, can comprise any type of XR device. For example, an XR device, as referred to herein, may be hardware that enables (e.g. user) interaction with immersive technologies (e.g. that blend a physical world with a digital world). An XR device may comprise one or more of a virtual reality (VR) device, augmented reality (AR) device, and a mixed reality (MR) device. Depending on its purpose, an XR device may be configured toprovide fully virtual experiences (e.g. by enhancing real-world views with digital overlays), or may be configured to blend physical and digital elements interactively.

[0033] Figure 1 is a schematic illustration of an existing XR architecture. As illustrated in Figure 1 , the architecture can comprise a system comprising a plurality of modules and / or components. The system can be adapted to provide a framework for creating portable, high-performance XR applications (e.g. that can run on different hardware and / or software environments). At least some or all of the modules (e.g. components) of the system illustrated in Figure 1 may be standardised by OpenXR. As such, the system illustrated in Figure 1 may be said to be an example of an OpenXR system.

[0034] As illustrated in Figure 1 , the system may be adapted to utilise a plurality of modules and / or components. In some examples, as illustrated in Figure 1, the system can comprise an application module 100, a loader module 102 (“Loader Trampoline”), and a runtime module 112. The loader module 102 can be configured to manage interaction between applications and underlying (e.g. OpenXR) runtime(s). The loader module 102 can be a dynamic loader module. For example, the loader module 102 can be configured to provide an abstraction layer. The abstraction layer can allow applications to interface with an appropriate (e.g. OpenXR) runtime (e.g. for a target platform). The loader module 102 can be configured to (e.g. help) simplify the development process by handling details of runtime selection and initialization, enabling applications to seamlessly work with different XR devices and platforms.

[0035] The runtime module 112 may be a software component. For example, the runtime module 112 may be a software component configured to (e.g. directly) interact with an underlying (e.g. XR device) hardware and / or manage the execution of an XR application. The runtime module 112 may be responsible for handling processes such as device enumeration, tracking, rendering, input handling, and / or other (e.g. low-level) operations required for delivering immersive experiences to XR device users. The runtime module 112 may be referred to as a OpenXR implementation module. Runtime modules (e.g. OpenXR runtimes) are typically provided by hardware vendors and / or platform providers. The runtime module 112 may be used to enable interoperability and / or performance of (e.g. XR) applications across different devices and / or platforms.

[0036] As illustrated in Figure 1, the system can comprise one or more optional modules 106, 108, 110, such as a loader terminator module 110. As also illustrated in Figure 1 , theone or more optional modules 106, 108, 110 can comprise one or more layer modules 106, 108. A layer module may be referred to herein as a layer, and vice-versa. OpenXR layers are a component of the OpenXR standard. Layers can, for example, be software components. In some examples, a layer may be, and / or may comprise, an API layer. In some cases, a layer can be configured to intercept and / or modify the behavior of the runtime module 112. As such, a layer can allow for various forms of customization and / or enhancement of an application. In OpenXR, layers can be used to implement features such as debugging, performance profiling, input device emulation, and other forms of runtime modification. By enabling developers to add custom functionality to OpenXR runtime, layers provide a flexible and extensible framework for building XR applications. As illustrated in Figure 1, the one or more layer modules 106, 108 may comprise two layer modules. However, it will be understood that this is merely an example, and that the one or more layer modules 106, 108 can comprise any number (e.g. three, four, five, etc.) of layer modules. As described herein, the one or more layer modules 106, 108 are optional. As such, it will be understood that the system of Figure 1 may comprise no layer modules according to some examples.

[0037] Through implementation of components such as those illustrated in Figure 1, OpenXR is able to provide a comprehensive framework for creating portable, high-performance XR applications that can run on diverse hardware and software environments. The standardization of these components helps to streamline development efforts and drive innovation in the field of XR.

[0038] Although OpenXR is attempting to homogenize the heterogeneity of XR devices, OpenXR does not currently support a mechanism for efficiently handling programs for XR devices. To better illustrate the current challenges associated with handling programs (e.g. applications) forXR devices, reference will now be made to an existing technique.

[0039] Figure 2 is a signalling diagram illustrating an exchange of signals in an environment according to an existing technique. As illustrated in Figure 2, the environment comprises a developer 200, an Integrated Development Environment (IDE) module 202, and a runtime module 204 (“OpenXR Runtime”). The IDE module 202 can be configured to receive (e.g. via upload) XR applications. In some cases, the IDE module can be used to generate code forXR applications. As illustrated in Figure 2, the runtime module 204 can be an OpenXR runtime module. However, it will be understood that this is merelyan example and that the runtime module 204 can comprise any type of (e.g. XR) runtime module, according to other examples.

[0040] As illustrated at block 206 of Figure 2, the developer 200 develops an application intended to be used on a plurality of XR devices. In the example of Figure 2, the plurality of XR devices can be said to include a total of N XR devices. As illustrated at block 206 of Figure 2, the application can be a to-do list application for displaying a to-do list (e.g. to a user) in XR. However, to account for device homogeneity, the developer 200 knows that they must manually adapt the application based on the display configuration (e.g. screen size, resolution, colour capabilities, etc.) of each XR device of the plurality of XR devices.

[0041] As illustrated at block 208 of Figure 2, the developer 200 can query a runtime module 204 (“OpenXR Runtime”) for display information (e.g. of each XR device of the plurality of XR devices). As illustrated by arrow 21 Oof Figure 2, the developer 200 (e.g. manually) can query the runtime module 204 for the display information. As illustrated by arrow 212 of Figure 2, the developer 200 obtains the display information from the runtime module 204. Although not illustrated in the example of Figure 2, it will be understood that the developer 200 may obtain the display information by other means. For instance, in some scenarios, the developer 200 may obtain the display information from an XR device datasheet. As illustrated at block 214 of Figure 2, the technique illustrated in Figure 2 requires the developer 200 to (i.e. manually) adapt the application to the specific display capabilities of each XR device of the plurality of XR devices based on the obtained display information.

[0042] As illustrated in Figure 2, the developer 200 must perform N adaptations of the application for each XR device of the plurality of XR devices. As illustrated by arrow 216 of Figure 2, the developer 200 can provide (e.g. upload) an adapted version of the application, to the IDE module 202, for each corresponding XR device of the plurality of XR devices. The developer 200 may use the IDE module 202 to generate each adapted version of the application. In some scenarios, the developer 200 may need to produce a different application for each XR device of the plurality of XR devices. As such, even when using existing techniques (e.g. such as those standardised by OpenXR) the developer 200 must still manually or programmatically adapt the application to capabilities of each XR device of the plurality of XR devices. As such, existing techniques such as that illustrated in Figure 2 are resource and energy inefficient.Figure 3 illustrates a first entity 10 in accordance with an embodiment. The first entity 10 can be for handling a program for an XR device. In some embodiments, the first entity 10 referred to herein can refer to equipment capable, configured, arranged and / or operable to communicate directly or indirectly with the second entity referred to herein, and / or with other nodes or equipment to enable and / or to perform the functionality described herein. In some embodiments, the first entity 10 referred to herein can, for example, be a physical node (e.g. a physical machine or server) or a virtual node (e.g. a virtual machine, VM). In some examples, the first entity 10 can comprise the first XR device referred to herein. In some examples, the first entity 10 may be the first XR device. In these examples, the processing circuitry 12 of the first entity 10 may be the processing circuitry of the first XR device. Alternatively, or in addition, in some examples, the first entity 10 may comprise the first API layer referred to herein.

[0043] As illustrated in Figure 3, the first entity 10 comprises processing circuitry (or logic) 12. The processing circuitry 12 controls the operation of the first entity 10 and can implement the method described herein in respect of the first entity 10. The processing circuitry 12 can be configured or programmed to control the first entity 10 in the manner described herein. The processing circuitry 12 can comprise one or more hardware components, such as one or more processors, one or more processing units, one or more multi-core processors and / or one or more modules. In particular implementations, each of the one or more hardware components can be configured to perform, or is for performing, individual or multiple steps of the method described herein in respect of the first entity 10. In some embodiments, the processing circuitry 12 can be configured to run software to perform the method described herein in respect of the first entity 10. The software may be containerised according to some embodiments. Thus, in some embodiments, the processing circuitry 12 may be configured to run a container to perform the method described herein in respect of the first entity 10.

[0044] Briefly, the processing circuitry 12 of the first entity 10 is configured to obtain, by a first API layer, a first request for initiation of an runtime system. The first request is requested by a first XR device. The processing circuitry 12 of the first entity 10 is configured to obtain, by the first API layer, first information indicative of one or more characteristics of the first XR device. The processing circuitry 12 of the first entity 10 is configured to obtain, by the first API layer, a second request for the runtime system to execute an original program. The original program is configured to be executed to generate anoutput at the first XR device. The processing circuitry 12 of the first entity 10 is configured to, in response to obtaining the second request, modify, by the first API layer, the original program based on the first information to generate a first modified program. The processing circuitry 12 of the first entity 10 is configured to provide, by the first API layer, the first modified program to the runtime system for execution of the modified program.

[0045] As illustrated in Figure 3, in some embodiments, the first entity 10 may optionally comprise a memory 14. The memory 14 of the first entity 10 can comprise a volatile memory or a non-volatile memory. In some embodiments, the memory 14 of the first entity 10 may comprise a non-transitory media. Examples of the memory 14 of the first entity 10 include, but are not limited to, a random access memory (RAM), a read only memory (ROM), a mass storage media such as a hard disk, a removable storage media such as a compact disk (CD) or a digital versatile disk (DVD), and / or any other memory.

[0046] The processing circuitry 12 of the first entity 10 can be communicatively coupled (e.g. connected) to the memory 14 of the first entity 10. In some embodiments, the memory 14 of the first entity 10 may be for storing program code or instructions which, when executed by the processing circuitry 12 of the first entity 10, cause the first entity 10 to operate in the manner described herein in respect of the first entity 10. For example, in some embodiments, the memory 14 of the first entity 10 may be configured to store program code or instructions that can be executed by the processing circuitry 12 of the first entity 10 to cause the first entity 10 to operate in accordance with the method described herein in respect of the first entity 10. Alternatively or in addition, the memory 14 of the first entity 10 can be configured to store any information, data, messages, requests, responses, indications, notifications, signals, or similar, that are described herein. The processing circuitry 12 of the first entity 10 may be configured to control the memory 14 of the first entity 10 to store any of the information, data, messages, requests, responses, indications, notifications, signals, or similar, that are described herein.

[0047] In some embodiments, as illustrated in Figure 3, the first entity 10 may optionally comprise a communications interface 16. The communications interface 16 of the first entity 10 can be communicatively coupled (e.g. connected) to the processing circuitry 12 of the first entity 10 and / or the memory 14 of the first entity 10. The communications interface 16 of the first entity 10 may be operable to allow the processing circuitry 12 of the first entity 10 to communicate with the memory 14 of the first entity 10 and / or vice versa. Similarly, the communications interface 16 of the first entity 10 may be operableto allow the processing circuitry 12 of the first entity 10 to communicate with any one or more nodes (e.g. the second entity) referred to herein and / or any other node. The communications interface 16 of the first entity 10 can be configured to transmit and / or receive any of the information, data, messages, requests, responses, indications, notifications, signals, or similar, that are described herein. In some embodiments, the processing circuitry 12 of the first entity 10 may be configured to control the communications interface 16 of the first entity 10 to transmit and / or receive any of the information, data, messages, requests, responses, indications, notifications, signals, or similar, that are described herein.

[0048] Although the first entity 10 is illustrated in Figure 3 as comprising a single memory 14, it will be appreciated that the first entity 10 may comprise at least one memory (i.e. a single memory or a plurality of memories) 14 that operate in the manner described herein. Similarly, although the first entity 10 is illustrated in Figure 3 as comprising a single communications interface 16, it will be appreciated that the first entity 10 may comprise at least one communications interface (i.e. a single communications interface or a plurality of communications interfaces) 16 that operate in the manner described herein. It will also be appreciated that Figure 3 only shows the components required to illustrate an embodiment of the first entity 10 and, in practical implementations, the first entity 10 may comprise additional or alternative components to those shown.

[0049] Figure 4 is a block diagram illustrating a method performed according to an embodiment. The method is computer-implemented. The method is for handling a program for an XR device. The first entity 10 described earlier with reference to Figure 3 can be configured to operate in accordance with the method of Figure 4. The method can be performed by or under the control of the processing circuitry 12 of the first entity 10 according to some examples.

[0050] With reference to Figure 4, as illustrated at block 302, a first request for initiation of a runtime system is obtained by a first application programming interface (API) layer. The first entity 10 (e.g. the processing circuitry 12 of the first entity 10) can be configured to cause the first API layer to obtain the first request. The first request is requested by a first XR device. As mentioned herein, in some examples, the first entity 10 may be the first XR device. In some examples, the first XR device may comprise the first API layer. In these examples, the first API layer may obtain the first request from another module (e.g. layer) and / or component of the first XR device. For example, in scenarios in whichthe first XR device comprises the first API layer, the first API layer may obtain the first request from a loader module of the first XR device. As mentioned herein, the first request is for initiation of the runtime system. The runtime system may be configured to execute one or more programs (e.g. applications) for the first XR device. The runtime system may comprise one or more hardware modules and / or one or more virtual modules. In some examples, the first XR device and / or the first entity 10 may comprise the runtime system. In some examples, at least a portion of the runtime system may be comprised in a cloud (e.g. network). The first request may be a request for the runtime system to initiate a (e.g. XR) runtime session for the first XR device. In some examples, the first request can be an indication that the first XR device requests initiation of XR functionality (e.g. in order to generate an output at the first XR device).

[0051] In some examples, the first request may be comprised in a system call. In these examples, obtaining the first request may comprise obtaining a system call comprising the first request. The system call may be forthe runtime system. Although not illustrated in Figure 4, in some examples, obtaining the first request may comprise intercepting the first request. For example, the first API layer may intercept the first request between the (e.g. processing circuitry of the) first XR device and the runtime system.

[0052] As illustrated at block 304 of Figure 4, first information is obtained by the first API layer. The first information is indicative of one or more characteristics of the first XR device. In some examples, the first information can be stored in a memory. For example, the first information can be stored in a memory of the first XR device. In examples in which the first XR device is the first entity 10, the first information may be stored in the memory 14 of the first entity 10. In some examples, the first information can be obtained (e.g. by the first API layer) from the memory of the first XR device. In some examples, the first information may be comprised (e.g. stored) in the runtime system. In some examples, obtaining the first information may comprise providing, by the first API layer, a third request to the runtime system. The third request can be a request for the first information. In some examples, the first information can be obtained (e.g. received) from the runtime system (e.g. in response to the third request).

[0053] In some examples, the one or more characteristics of the first XR device may comprise one or more hardware characteristics of the first XR device. Alternatively, or in addition, in some examples, the one or more characteristics may comprise one or more functional characteristics of the first XR device. In some examples, the one or more characteristicsof the first XR device may comprise one or more of a type of the first XR device, a form factor of the first XR device, a type of a display of the first XR device, a size of the display of the first XR device, a display resolution of the display of the first XR device, a type of sensor of the first XR device, and an interaction mechanism of the first XR device.

[0054] As mentioned above, in some examples, the one or more characteristics of the first XR device may comprise a type of a display of the first XR device. Different XR devices can have a broad spectrum of display sizes and display resolutions. The first XR device may be, for example, a compact headset comprising one or more small (e.g. display) screens. In some examples, the first XR device may be an immersive device comprising high-resolution displays (e.g. that cover a user's entire field of view). The variation in display types, sizes and resolutions among XR devices can impact visual fidelity, spatial immersion, and / or overall user experience ofXR applications.

[0055] In some examples, the display type of the first XR device may be indicative of the number and / or type of displays comprised in the first XR device. For example, the display type of the first XR device may indicate that the first XR device comprises a display screen. The display screen may be any type of display screen. For example, the first XR device may comprise one or more of a liquid-crystal display (LCD), an organic light-emitting diode (OLED) display, and / or a micro LED. The display type of the first XR device may comprise one or more attributes of a display of the first XR device, such as pixel density, refresh rate, colour accuracy, and / or contrast ratio. In some examples, the display type of the first XR device may be indicative of a capability of the one or more displays of the first XR device. For example, the display type of the first XR device may indicate whether the one or more displays of the first XR device are configured to perform eye-tracking, foveated rendering, and / or variable focus display.

[0056] XR devices can be configured with different sensor technologies. As mentioned herein, the one or more characteristics of the first XR device can include a type of sensor of the first XR device. For example, the type of sensor can comprise an indication of sensor model and / or a type (e.g. range) of data measured by the sensor. In some examples, the type of sensor can be indicative of a camera type, a camera model, and / or a camera resolution. In some examples, the type of sensor of the first XR device can comprise an inertial motion unit (IMU) sensor, a Light Detection and Ranging (LiDAR) sensor, and / or a red, green, and blue (RGB) camera.As also mentioned herein, in some examples, the one or more characteristics of the first XR device can include an interaction mechanism of the first XR device. The interaction mechanism of the first XR device may be indicative of an input that the first XR device is configured to receive, and / or an output that the first XR device is configured to provide (e.g. generate). For example, the interaction mechanism of the first XR device may be indicative of whether the first XR device is configured to perform hand tracking, gesture recognition, motion recognition, gaze-based interaction, and / or voice command recognition. In some examples, the one or more characteristics of the first XR device can comprise an indication of one or more input types supported by the first XR device. The one or more input types may comprise a physical (e.g. Bluetooth) keyboard, and / or a physical mouse. In some examples, the one or more input types may comprise a virtual soft keyboard (e.g. controlled by hand gestures). In some examples, the interaction mechanism of the first XR device can comprise one or more haptic actuators. In these examples, the one or more characteristics of the first XR device may comprise an availability of the one or more haptic actuators and / or a haptic actuator frequency range.

[0057] In some examples, the one or more characteristics of the first XR device may include a tracking system of the first XR device. XR devices can utilize diverse tracking systems. In some examples, the tracking system of the first XR device may comprise an inside-out tracking system, an outside-in tracking system, and / or a hybrid tracking system (e.g. for head and / or hand tracking). An inside-out tracking system may, for example, comprise one or more (e.g. infrared (IR)) cameras. An outside-in tracking system may comprise one or more IR reflectors in combination with one or more (e.g. external) IR lighthouses. The one or more characteristics of the first XR device may be indicative of the capability of the tracking system of the first XR device. A capability of the tracking system may include spatial mapping capability, environmental understanding capability, and / or room-scale tracking capability.

[0058] As mentioned herein, the one or more characteristics of the first XR device can comprise one or more hardware characteristics of the first XR device. For example, the one or more characteristics of the first XR device can comprise a form factor of the first XR device, and / or an ergonomic configuration of the first XR device. The form factor of the ergonomic device may include a (e.g. standalone) headset, a tethered headsets, mixed reality glasses, a smartphone and / or a tablet. The one or more hardware characteristics of the first XR device may be indicative of the first XR device’s weight, comfort, fit,portability impact, comfort level, freedom of movement, and / or overall wearability. The design of such hardware characteristics can impact the inclusivity and comfort of XR experience provided by the first XR device.

[0059] As illustrated at block 306 of Figure 4, a second request is obtained by the first API layer. The second request is for the runtime system to execute an original program. The original program is configured to be executed to generate an output at the first XR device. The original program may comprise, for example, an (e.g. XR) application. In some examples, the output can comprise one or more of a visual output, a haptic output, and an audio output. For example, the original program may be configured to be executed to generate a display at the first XR device (e.g. via a screen of the first XR device). In some examples, the original program can be stored in a memory of the first XR device. Although not illustrated in Figure 4, in some examples, the original program may be obtained, by the first API layer, from the memory of the first XR device. In some examples, obtaining the second request, as defined herein, may comprise intercepting the second request. For example, the first API layer may intercept the second request between the (e.g. processing circuitry of the) first XR device and the runtime system.

[0060] As illustrated at block 308 of Figure 4, in response to obtaining the second request, the original program is modified by the first API layer to generate a modified program. The original program is modified based on the first information. That is, the original program is modified based on the first information that is indicative of one or more characteristics of the first XR device, as defined herein. In this way, the original program can be modified to account for the one or more characteristics of the first XR device. For example, the original program can be modified to adapt to different display sizes and / or display resolutions of the first XR device to ensure optimal visual quality and readability. Alternatively or in addition, the original program may be modified to accommodate different input types of the first XR device. In this way, the modified program can be a version of the original program that is optimized for the first XR device. In some examples, modifying the original program can comprise modifying at least a portion of code of the original program. For example, modifying the original program may comprise modifying (e.g. adapting) only some of the code associated with the original program (i.e. to generate the modified program). In some examples, modifying the original program can comprise modifying one or more software abstractions of the original program. The modified program may be configured to generate the output associated with the originalprogram in a manner that is customized to the one or more characteristics of the first XR device, as described herein.

[0061] As illustrated at block 310, in response to obtaining the second request, the modified program is provided, by the first API layer, to the runtime system for execution of the modified program. In some examples, providing the modified program to the runtime system can comprise providing the modified program instead of the original program.

[0062] In some examples, the original program can be a device agnostic program. For example, the original program may not be configured to be executed in respect of a particular (e.g. XR) device. The original program may comprise generic code (e.g. to be provided to the runtime system of a plurality of XR devices) and the modified code may comprise specific code (e.g. a version of the generic code that is modified specifically for a particular XR device, such as the first XR device referred to herein). In some examples, the original program may be created using a second API layer. In some examples, both the first API layer and the second API layer referred to herein may be associated with a (e.g. single) set of corresponding API layers. For example, the first API layer and the second API layer may be linked (e.g. by belonging to a corresponding set of API layers). In some examples, the original program can comprise at least one information element (e.g. piece of code) indicative of a requirement to modify the original program. In some examples, modifying the original program (i.e. by the first API layer) may comprise modifying the original program in response to identifying the at least one information element. Therefore, in some examples, the first API layer may identify the at least one information element in the original program.

[0063] The first XR device referred to herein may be any type of XR device. For example, the first XR device may comprise one or more of an augmented reality (AR) device, a virtual reality (VR) device, and a mixed reality (MR) device. In some examples, the first XR device may comprises one or more of a user equipment (UE), a wearable device, a headset, glasses, a television, a smartphone, and a tablet. In some examples, the runtime system referred to herein may be at least partially comprised in the first XR device. Alternatively, or in addition, the runtime system referred to herein may be at least partially comprised in a cloud network. In some examples, the runtime system (e.g. a runtime module) may be comprised in the first XR device as a component of the first XR device. In some examples, the runtime system may be comprised in an entity that is different to the first XR device.Figure 5 illustrates a second entity 20 in accordance with an embodiment. The second entity 20 is for handling a program for an XR device. In some embodiments, the second entity 20 referred to herein can refer to equipment capable, configured, arranged and / or operable to communicate directly or indirectly with the first entity 10 referred to herein, and / or with other nodes or equipment to enable and / or to perform the functionality described herein. In some embodiments, the second entity 20 referred to herein can, for example, be a physical node (e.g. a physical machine or server) or a virtual node (e.g. a virtual machine, VM). In some examples, the second entity 20 can comprise the first XR device referred to herein. In some examples, the second entity 20 may be the first XR device. In these examples, the processing circuitry 22 of the second entity 20 may be the processing circuitry of the first XR device.

[0064] As illustrated in Figure 5, the second entity 20 comprises processing circuitry (or logic) 22. The processing circuitry 22 controls the operation of the second entity 20 and can implement the method described herein in respect of the second entity 20. The processing circuitry 22 can be configured or programmed to control the second entity 20 in the manner described herein. The processing circuitry 22 can comprise one or more hardware components, such as one or more processors, one or more processing units, one or more multi-core processors and / or one or more modules. In particular implementations, each of the one or more hardware components can be configured to perform, or is for performing, individual or multiple steps of the method described herein in respect of the second entity 20. In some embodiments, the processing circuitry 22 can be configured to run software to perform the method described herein in respect of the second entity 20. The software may be containerised according to some embodiments. Thus, in some embodiments, the processing circuitry 22 may be configured to run a container to perform the method described herein in respect of the second entity 20.

[0065] Briefly, the processing circuitry 22 of the second entity 20 is configured to provide, by a first XR device, a first request for initiation of a runtime system. The processing circuitry 22 of the second entity 20 is configured to provide, by the first XR device, a second request for the runtime system to execute an original program. The original program is configured to be executed to generate an output at the first XR device. The processing circuitry 22 of the second entity 20 is configured to, in response to providing the second request, obtain, by the first XR device, one or more instructions from the runtime system.The one or more instructions are generated by execution of a modified program. The modified program corresponds to a modified version of the original program. The original program is modified, by a first API layer, based on first information indicative of one or more characteristics of the first XR device. The processing circuitry 22 of the second entity 20 is configured to execute, by the first XR device (602), the one or more instructions.

[0066] As illustrated in Figure 5, in some embodiments, the second entity 20 may optionally comprise a memory 24. The memory 24 of the second entity 20 can comprise a volatile memory or a non-volatile memory. In some embodiments, the memory 24 of the second entity 20 may comprise a non-transitory media. Examples of the memory 24 of the second entity 20 include, but are not limited to, a random access memory (RAM), a read only memory (ROM), a mass storage media such as a hard disk, a removable storage media such as a compact disk (CD) or a digital versatile disk (DVD), and / or any other memory.

[0067] The processing circuitry 22 of the second entity 20 can be communicatively coupled (e.g. connected) to the memory 24 of the second entity 20. In some embodiments, the memory 24 of the second entity 20 may be for storing program code or instructions which, when executed by the processing circuitry 22 of the second entity 20, cause the second entity 20 to operate in the manner described herein in respect of the second entity 20. For example, in some embodiments, the memory 24 of the second entity 20 may be configured to store program code or instructions that can be executed by the processing circuitry 22 of the second entity 20 to cause the second entity 20 to operate in accordance with the method described herein in respect of the second entity 20. Alternatively or in addition, the memory 24 of the second entity 20 can be configured to store any information, data, messages, requests, responses, indications, notifications, signals, or similar, that are described herein. The processing circuitry 22 of the second entity 20 may be configured to control the memory 24 of the second entity 20 to store any of the information, data, messages, requests, responses, indications, notifications, signals, or similar, that are described herein.

[0068] In some embodiments, as illustrated in Figure 5, the second entity 20 may optionally comprise a communications interface 26. The communications interface 26 of the second entity 20 can be communicatively coupled (e.g. connected) to the processing circuitry 22 of the second entity 20 and / or the memory 24 of the second entity 20. Thecommunications interface 26 of the second entity 20 may be operable to allow the processing circuitry 22 of the second entity 20 to communicate with the memory 24 of the second entity 20 and / or vice versa. Similarly, the communications interface 26 of the second entity 20 may be operable to allow the processing circuitry 22 of the second entity 20 to communicate with any one or more nodes (e.g. first entity) referred to herein and / or any other node. The communications interface 26 of the second entity 20 can be configured to transmit and / or receive any of the information, data, messages, requests, responses, indications, notifications, signals, or similar, that are described herein. In some embodiments, the processing circuitry 22 of the second entity 20 may be configured to control the communications interface 26 of the second entity 20 to transmit and / or receive any of the information, data, messages, requests, responses, indications, notifications, signals, or similar, that are described herein.

[0069] Although the second entity 20 is illustrated in Figure 5 as comprising a single memory 24, it will be appreciated that the second entity 20 may comprise at least one memory (i.e. a single memory or a plurality of memories) 24 that operate in the manner described herein. Similarly, although the second entity 20 is illustrated in Figure 5 as comprising a single communications interface 26, it will be appreciated that the second entity 20 may comprise at least one communications interface (i.e. a single communications interface or a plurality of communications interfaces) 26 that operate in the manner described herein. It will also be appreciated that Figure 5 only shows the components required to illustrate an embodiment of the second entity 20 and, in practical implementations, the second entity 20 may comprise additional or alternative components to those shown.

[0070] Figure 6 is a block diagram illustrating a method performed according to an embodiment. The method is computer-implemented. The method is for handling a program for an XR device. The second entity 20 described earlier with reference to Figure 5 can be configured to operate in accordance with the method of Figure 6. The method can be performed by or under the control of the processing circuitry 22 of the second entity 20 according to some examples.

[0071] With reference to Figure 6, as illustrated at block 402, a first request for initiation of a runtime system is provided by a first XR device. For example, the processing circuitry of the first XR device (e.g. the processing circuitry 22 of the second entity 20) can provide the first request. The first request may be provided to the runtime system and / or the first API layer, as defined herein. For example, the first XR device may initiate transmissionof the first request towards the runtime system via the first API layer. The first request can be as described herein (e.g. with reference to Figure 4).

[0072] As illustrated at block 404 of Figure 6, a second request is provided by the first XR device. In some examples, the processing circuitry of the first XR device (e.g. the processing circuitry 22 of the second entity 20) can provide the second request. The second request may be provided to the runtime system and / or the first API layer, as defined herein. For example, the first XR device may initiate transmission of the second request towards the runtime system via the first API layer. The second request is a request for the runtime system (608) to execute an original program. The original program is configured to be executed to generate an output at the first XR device, as defined herein.

[0073] As illustrated at block 406 of Figure 6, in response to providing the second request, one or more instructions are obtained, by the first XR device, from the runtime system. In some examples, the processing circuitry of the first XR device (e.g. the processing circuitry 22 of the second entity 20) can obtain the one or more instructions. The one or more instructions are generated by execution of a modified program. The modified program corresponds to a modified version of the original program. The original program is modified, by the first API layer, based on first information indicative of one or more characteristics of the first XR device, as defined herein. In some examples, the runtime system, as referred to herein, may execute the modified program to generate the one or more instructions.

[0074] As illustrated at block 408 of Figure 6, the one or more instructions are executed by the first XR device (602). The one or more instructions can comprise one or more instructions for generating the (e.g. modified) output at the first XR device. In some examples, executing the one or more instructions may comprise generating the output, by the first XR device (602), based on the one or more instructions. The output generated based on the one or more instructions may be an XR output that is customised to account for the one or more characteristics of the first XR device, as defined herein.

[0075] Although not illustrated in Figure 6, in some examples, providing the first request may comprise providing a system call comprising the first request. For example, the (e.g. processing circuitry) of the first XR device may provide the system call to the runtime system and / or the first API layer. As described herein, the original program may be stored in a memory of the first XR device. Although also not illustrated in Figure 6, insome examples, the original program may be obtained (e.g. retrieved), by the (e.g. processing circuitry of the) first XR device, from the memory of the first XR device.

[0076] There is also provided a system comprising the first entity 10 described herein and the second entity 20 described herein. A method performed by the system can comprise the method described herein in respect of the first entity 10 and / or the method described herein in respect of the second entity 20 in a compatible way.

[0077] Thus, in the manner described herein the first API Layer can be configured to adapt an original XR program to the capabilities of specific XR devices. As mentioned herein, in some examples, the original program may be created using a second API layer, as defined herein. The second API layer may be configured to provide access to a developer library. The developer library can comprise (e.g. store) abstractions for creating XR assets (e.g. an XR canvas). The developer library may be embedded as an extension of an existing application-runtime interface layer (e.g. a OpenXR API). Alternatively, the developer library may be implemented as a wrapper library on top of an existing (e.g. OpenXR) XR architecture. If a developer desires XR components (e.g. of an application) to adapt to device capabilities, they can use the second API layer, as defined herein (e.g. instead of an API provided by the development platform being used by the developer).

[0078] When an original program (e.g. application) is deployed to a specific XR device, the first API layer described herein can adapt the (e.g. XR components of the) original program with respect to the specific capabilities of the specific XR device. For example, if the original program is configured (e.g. by a developer) to be executed to display a to-do list (e.g. comprising a plurality of text information elements) at an XR device, the first API layer may translate the original program into a modified program depending on whether the XR device is a highly capable device or a lower-end device. In examples in which the XR device is a highly capable device, the modified program may be configured to be executed to display a (e.g. fancy) 3D model. In examples in which the XR device is a lower-end device, the modified program may be configured to be executed to display a simple 2D frame (e.g. with bullet items). In this way, developers do not need to write additional code to target multiple XR devices with different capabilities.

[0079] Figure 7 is a signalling diagram illustrating an exchange of signals in a system according to an embodiment. As illustrated in Figure 7, the system may comprise a first XR device602 (“Capable device”) as defined herein. As also illustrated in Figure 7, the system may comprise a first API layer 606 (“New API layer”), as defined herein, and a runtime system 608 (“Runtime”), as defined herein. As illustrated in Figure 7, in some examples, the system may comprise a second XR device 604 (“Constrained Device”). The second XR device may have one or more characteristics that are different to the one or more characteristics of the first XR device 602, as defined herein. For example, the first XR device 602 may comprise a display with 3D display capability, and the second XR device may comprise a display without 3D display capability.

[0080] As illustrated in Figure 7, in some examples, the first XR device 602, the first API layer 606, and the runtime system 608 may be separate entities. Alternatively, in some examples, the first entity 10 as described herein may comprise the first API layer 606, the second entity 20 as described herein may comprise the first XR device 602, and the first entity 10 and the second entity 20 may be separate entities. It will be understood that the system illustrated in Figure 7 is merely an example, and that other configurations of the first XR device 602, the first API layer 606, and the runtime system 608 are possible. For example, the first XR device 602, the first API layer 606, and the runtime system 608 may be comprised in the same entity. In some examples, the first XR device 602 may comprise the first API layer and / or the runtime system 608. For example, the first XR device 602 may be the second entity 20 as described herein, the first API layer 606 may be the first entity 10 as described herein, and the first XR device 602 (e.g. second entity 20) may comprise the first API layer 606 (e.g. first entity 10). Alternatively, the first API layer 606 and / or the runtime system 608 may be comprised in a cloud network.

[0081] As illustrated at block 610 of Figure 7, a developer 600 may create an original program (e.g. XR application). As also illustrated in Figure 7, the original program may be configured to be executed to generate an output. As illustrated in Figure 7, in some examples, the output may be a display of a to-do list on a head-mounted display. It will be understood that this is merely an example, and that the original program may be configured to be executed to generate any type of (e.g. XR) output. In some examples, the original program may be created using a second API layer, as defined herein. For example, the original program (e.g. application code) may be written using the second API layer.As illustrated by arrow 612 of Figure 7, the first XR device 602 may obtain the original program. For example, the developer 600 may upload the original program to the first XR device 602. The first XR device 602 may store the original program in a memory of the first XR device 602, as described herein. As illustrated by arrow 614 of Figure 7, in some examples, the second XR device 604 may obtain the original program. For example, the developer 600 may upload the original program to the second XR device 604. The second XR device 604 may store the original program in a memory of the second XR device 604. The original program obtained by the first XR device 602 can comprise the same code as the original program obtained by the second XR device 604. Therefore, in some examples, the developer 600 may upload the same original program (e.g. application) to at least two XR devices.

[0082] As illustrated at block 616 of Figure 7, in some examples, the first API layer 606 may be configured to read information from the runtime system 608 in response to initialisation of a session for an XR device. As illustrated by arrow 618 of Figure 7, a first request (“InitSessionO”) for initiation of the runtime system 608 is provided by the first XR device 602, as described herein. The first request may be provided in response to (e.g. an attempt to) running the original program, by the first XR device 602. The first request may be a request to initialise a (e.g. OpenXR) runtime session for the first XR device 602. As also illustrated by arrow 618 of Figure 7, the first API layer 606 obtains the first request. In some examples, as illustrated in Figure 7, the first API layer 606 can intercept the first request, as defined herein.

[0083] As illustrated by arrow 620 of Figure 7, the first API layer 606 may provide (e.g. forward) the first request to the runtime system 608. As such, in some examples, the runtime system 608 can obtain (e.g. receive) the first request from the first API layer 606. Although not illustrated in Figure 7, the runtime system 608 may initiate (e.g. a session for the first XR device 602) in response to obtaining the first request.

[0084] As illustrated by arrow 622 of Figure 7, the first API layer 606 may provide a third request (“QuerySysteminfo()”) to the runtime system 608. The third request can be a request for first information, as defined herein. In some examples, the first request and the third request may be provided (e.g. by the first API layer 606) simultaneously. For example, the method steps as described with respect to arrows 620 and 622 of Figure 7 can be performed by the first API layer 606 providing (e.g. transmitting) a (e.g. single) first message, comprising the first request and the third request, to the runtime system 608.The first information may comprise system information associated with the first XR device 602. As illustrated by arrow 624 of Figure 7, the first API layer 606 obtains the first information (“Systeminfo”) indicative of one or more characteristics of the first XR device 602, as defined herein. For example, the first API layer 606 can receive the first information from the runtime system 608. The first API layer 606 may determine (e.g. infer) one or more capabilities of the first XR device 602 based on the first information. As illustrated by arrow 626 of Figure 7, the first API layer 606 may provide information indicative that the first request is successful to the first XR device 602. For example, the information may indicate that a runtime session has been initiated for the first XR device 602.

[0085] As illustrated at block 628 of Figure 7, in some examples, the steps as described with reference to arrows 618 to 626 of Figure 7 may be performed in respect of the second XR device 604 (i.e. instead of the first XR device 602).

[0086] As illustrated by arrow 630 of Figure 7, the first XR device provides a second request (“DisaplyToDoO”) for the runtime system 608 to execute the original program, as defined herein. As also illustrated by arrow 630 of Figure 7, the first API layer obtains the second request. In some examples, as illustrated in Figure 7, the first API layer 606 may intercept the second request. In some examples, the second request may comprise the original program, as defined herein. As illustrated at block 632 of Figure 7, the first API layer 606 can be configured to modify (e.g. adapt) the original program based on the first information, as defined herein. The second request may be comprised in a function call provided by the (e.g. processing circuitry of the) first XR device 602. As illustrated by arrow 634 of Figure 7, the first API layer 606 modifies the original program based on the first information to generate a modified program, as defined herein. For example, the first API Layer 606 may intercept the second request (e.g. comprised in the function call) and adapt the content of the original program (e.g. configured to be executed to generate a to-do list on an XR device display) to match the capabilities of the first XR device 602. For instance, the original program may be adapted based on the screen resolution of a display of the first XR device 602. In this example, the modified program may be configured to be executed to use a frame (e.g. for the to-do list) that fits half of a display (e.g. viewport) of the first XR device 602 (e.g. presented to a user).As illustrated by arrow 636 of Figure 7, the first API layer 606 provides the modified program (“DisplayToDeviceO”), as defined herein, to the runtime system 608 for execution of the modified program. Thus, in some examples, the runtime system 608 may obtain the modified program from the first API layer 606. In some examples, providing the modified program to the runtime system 608 may comprise providing the modified program instead of the original program, as defined herein, to the runtime system 608. Although not explicitly illustrated in Figure 7, the runtime system 608 may execute the modified program to generate one or more instructions. The one or more instructions may be for generating the output based on the one or more characteristics (e.g. type of display) of the first XR device (e.g. specifically), as defined herein.

[0087] As illustrated by arrow 638 of Figure 7, in some examples, the runtime system 608 can provide the one or more instructions (“ToDo”) to the (e.g. processing circuitry) of the first XR device 602. As also illustrated by arrow 638, the first XR device 602 obtains the one or more instructions from the runtime system 608. Although not explicitly illustrated in Figure 7, the first XR device 602 executes the one or more instructions. In some examples, executing the one or more instructions can comprise (e.g. cause) the first XR device 602 generating the output (e.g. via a user interface of the first XR device 602), as defined herein. For example, executing the one or more instructions can cause the first XR device 602 to generate the output associated with the modified program.

[0088] As illustrated at block 640 of Figure 7, in some examples, the steps as described with reference to arrows 630 to 638 may be performed in respect of the second XR device 604 (i.e. instead of the first XR device 602). In these examples, the modification of the original program based on the one or more characteristics of the first XR device 602 can be different to the modification of the original program based on one or more characteristics of the second XR device 604 (e.g. if the one or more characteristics of the first XR device 602 are different to the one or more characteristics of the second XR device 604).

[0089] In the example illustrated in Figure 7, the original program can be configured to be executed to show a user of an XR device a to-do list of items. The to-do list items may correspond to a plurality of information elements displayed as text. The to-do list items may describe actions and / or tasks to be completed by the user of the XR device. The to-do list may, for example, be for performing preventive maintenance at a piece of machinery on a factory floor. As an illustrative example, the first XR device 602 maycorrespond to a high-end device with 3D display capabilities. In such a scenario, the modification (e.g. adaptation) of the original program may include generating a modified program that is configured to be executed to display a high-definition 3D frame on a user interface of the first XR device 602. In this scenario, to-do items of the to-do list may be displayed in 3D text. As another example, the second XR device 604 may correspond to a low-end device without 3D display capability. In such a scenario, the modification (e.g. adaptation) of the original program may include generating a modified program that is configured to be executed to display a monochromatic 2-pixel-wide frame. In this scenario, the to-do items of the to-do list may be displayed in flat (e.g. non 3D) text. However, it will be understood that this is merely an example, and that the original program can be configured to generate other types of output at an XR device.

[0090] As described herein, in some examples, the original program may be created (e.g. generated) using a second API layer. An example of at least a portion of code of an original program is as follows:

[0091] var my_todo = invention . ToDo ( )

[0092] my_todo . Add ("Open the rack door")

[0093] my_todo . Add ("Clean the server fan")

[0094] my_todo . Add ("Close the rack door")

[0095] my_todo . Show ( )

[0096] As illustrated by the example above, the original program can comprise code and / or pseudo-code. As described herein, the original program can comprise at least one information element that is indicative of a requirement to modify the original program. For example, with reference to the above example of the original program, the at least one first information element can be a section of code and / or pseudo code (e.g. “invention. ToDoO”).

[0097] As mentioned herein, the first API layer 606 modifies the original program based on first information, as defined herein. Below is an example of pseudo-code describing how at least some of (e.g., the code of) the original program can be modified based on the first information. More specifically, the pseudo-code below provides an example of how the original program may be modified based on the screen resolution of an XR device.

[0098] var screen_resolution = OpenXR. GetHMDInfo ( )

[0099] if screen_resolution == (1920 , 1080 ) :

[0100] / / render high-resolution ToDoelse if screen_resolution == ( 800 , 600 ) :

[0101] / / render low-resolution ToDo

[0102] In some examples, some or all of the entities of the system illustrated in Figure 7 may be comprised in a network. The network can be any type of network. For example, the network may be a communications or telecommunications network. In some embodiments, the network can be a mobile network, such as a fifth generation (5G) mobile network or any other generation mobile network (e.g. 6G). In some examples, the network can be a core network (e.g. a 5G core (5GC) network) or a radio access network (RAN). In some examples, the network can be a virtual network or an at least partially virtual network. Although some examples have been provided for the type of network, it will be understood that the network referred to herein can be any other type of network.

[0103] Figure 8 is a block diagram illustrating an XR architecture according to an embodiment. The XR architecture illustrated in Figure 8 may be considered to represent an execution flow of an XR application. As illustrated in Figure 8, the XR architecture can comprise a plurality of modules 800, 804, 806, 810, 812. As also illustrated in Figure 8, the plurality of modules 800, 804, 806, 810, 812 can comprise an application module 800, a loader module 804, a first layer module 806, a second layer module 810, and a runtime module 812. The runtime module 812 can comprise the runtime system, as referred to herein. One or more of the plurality of modules 800, 804, 806, 810, 812 may be an OpenXR module (e.g. conform to OpenXR standards). As illustrated in Figure 8, in some examples, the XR architecture can comprise a first layer module 806 and a second layer module 810. In some examples, the XR architecture may only comprise a single layer module (e.g. the first layer module 806). For example, in some scenarios, the XR architecture may not comprise the second layer module 810. In some examples, the XR architecture may comprise additional layer modules to those illustrated in Figure 8 (e.g. a third layer module, a fourth layer module, a fifth layer module, etc.).

[0104] As illustrated in Figure 8, in some examples, the application module 800 may invoke a function (“xrFunction”). The function may be referred to herein as a program. The function may be provided by one or more (e.g. OpenXR) APIs. The function may traverse several modules (e.g. layers) of the XR architecture (e.g. stack). For example, the function can be invoked by the loader module 804, and then pass through one or moreof the first layer module 806 and the second layer module 810, before being executed by the runtime module 812.

[0105] Figure 8 illustrates an example manner in which a first API layer 804, as defined herein, may be implemented in the XR architecture. For example, the first API layer 804 may be comprised in the first layer module 806. When the function (e.g. original program, as defined herein) reaches the first API layer 804 (e.g. of the first layer module 806), the first API layer 804 may intercept the function and modify (e.g. adapt) the (e.g. code of the) function based on first information of a first XR device, as defined herein.

[0106] As mentioned herein, the original program, as defined herein, may be created using a second API layer. Figure 8 illustrates an example manner in which the second API layer 802, as defined herein, may be implemented in the XR architecture. For example, as illustrated in Figure 8, the second API layer 802 may be comprised in the application module 800. The second API layer 802 can expose an API to a (e.g. XR) developer for creating XR experiences. The second API layer can be configured to enable creation of an original program that can be automatically adapted to capabilities of an XR device (e.g. via the first API layer 804). The second API layer 802 may be implemented as part of an API layer in an XR architecture (e.g. such as that illustrated in Figure 8). Alternatively, in some examples, the second API layer may be implemented as an external module (e.g. library). The external module may be connected to (e.g. one or more modules) of the XR architecture.

[0107] There is also provided a computer program comprising instructions which, when executed by processing circuitry (such as the processing circuitry 12 of the first entity 10 described herein, and / or the processing circuitry 22 of the second entity 20 described herein), cause the processing circuitry to perform at least part of the method described herein. There is provided a computer program product, embodied on a non-transitory machine-readable medium, comprising instructions which are executable by processing circuitry (such as the processing circuitry 12 of the first entity 10 described herein, and / or the processing circuitry 22 of the second entity 20 described herein) to cause the processing circuitry to perform at least part of the method described herein. There is provided a computer program product comprising a carrier containing instructions for causing processing circuitry (such as the processing circuitry 12 of the first entity 10 described herein, and / or the processing circuitry 22 of the second entity 20 described herein) to perform at least part of the method described herein. In some embodiments,the carrier can be any one of an electronic signal, an optical signal, an electromagnetic signal, an electrical signal, a radio signal, a microwave signal, or a computer-readable storage medium.

[0108] In some embodiments, the first entity 10 functionality described herein, and / or the second entity 20 functionality described herein can be performed by hardware. Thus, in some embodiments, the first entity 10 described herein, and / or the second entity 20 described herein can be a hardware entity. However, it will also be understood that optionally at least part or all of the first entity 10 described herein, and / or the second entity 20 described herein can be virtualised. For example, the functions performed by the first entity 10 described herein, and / or the second entity 20 described herein can be implemented in software running on generic hardware that is configured to orchestrate the first entity functionality described herein, and / or the second entity 20 functionality described herein. Thus, in some embodiments, the first entity 10 described herein, and / or the second entity 20 described herein can be a virtual entity. Virtualised instances of the first entity 10, and / or the second entity 20 herein can be rapidly deployed, scaled, and / or modified according to changing needs of a network environment.

[0109] Therefore, as described herein, there are provided improved techniques and apparatus for handling a program for an XR device. The techniques described herein enable an original (e.g. generic) program for an XR device to be adapted based on the actual capabilities of the XR device which attempts to execute the original program. As such, the techniques described herein can be advantageously implemented to avoid the challenges associated with XR device heterogeneity. Some of the techniques described herein provide for software abstractions (e.g. APIs) that allow a developer to include components in their program (application), such as Ul components, that can be automatically modified (e.g. adapted) based on specific XR device capabilities, without the need for the developer to write device-specific code. The techniques described herein provide more efficient processes for developing XR applications, reduce resource usage (e.g. through both development time and software maintenance), and improve developer experience. Moreover, the techniques described herein are adaptable and can be implemented using OpenXR technology, and / or other application-runtime (e.g. API-layer) standard technology.

[0110] It should be noted that the above-mentioned embodiments illustrate rather than limit the idea, and that those skilled in the art will be able to design many alternative embodimentswithout departing from the scope of the appended claims. The word “comprising” does not exclude the presence of elements or steps other than those listed in a claim, “a” or “an” does not exclude a plurality, and a single processor or other unit may fulfil the functions of several units recited in the claims. Any reference signs in the claims shall not be construed so as to limit their scope.

Claims

CLAIMS1. A computer-implemented method for handling a program for an extended reality, XR, device, the method comprising:obtaining (302, 618), by a first application programming interface, API, layer (606), a first request for initiation of a runtime system (608), wherein the first request is requested by a first XR device (602);obtaining (304, 624), by the first API layer (606), first information indicative of one or more characteristics of the first XR device (602);obtaining (306, 630), by the first API layer (606), a second request for the runtime system (608) to execute an original program, wherein the original program is configured to be executed to generate an output at the first XR device (602);in response to obtaining (306, 630) the second request:modifying (308, 634), by the first API layer (606), the original program based on the first information to generate a modified program; and providing (310, 636), by the first API layer (606), the modified program to the runtime system (608) for execution of the modified program.

2. The method as claimed in any of the preceding claims, wherein obtaining (302, 618) the first request comprises intercepting the first request.

3. The method as claimed in any of the preceding claims, wherein obtaining (302, 618) the first request comprises obtaining a system call comprising the first request.

4. The method as claimed in any of the preceding claims, wherein obtaining (306, 630) the second request comprises intercepting the second request.

5. The method as claimed in any of the preceding claims, wherein:the first information is stored in a memory of the first XR device (602); and / or the original program is stored in a memory of the first XR device (602).

6. The method as claimed in claim 5, wherein the first information is obtained from the memory of the first XR device (602).

7. The method as claimed in claim 5 or 6, the method comprising:obtaining, by the first API layer (606), the original program from the memory ofthe first XR device.

8. The method as claimed in any of the preceding claims, wherein obtaining (304, 624) the first information comprises:providing, by the first API layer (606), a third request to the runtime system (608), wherein the third request is a request for the first information; andobtaining, by the first API layer (606), the first information from the runtime system (608).

9. The method as claimed in any of the preceding claims, wherein the runtime system (608) is at least partially comprised in:the first XR device (602); and / ora cloud network.

10. The method as claimed in any of the preceding claims, wherein the one or more characteristics of the first XR device (602) comprise:one or more hardware characteristics of the first XR device (602); and / or one or more functional characteristics of the first XR device (602).

11. The method as claimed in any of the preceding claims, wherein the one or more characteristics of the first XR device (602) comprise one or more of:a type of the first XR device (602);a form factor of the first XR device (602);a type of a display of the first XR device (602);a size of the display of the first XR device (602);a display resolution of the display of the first XR device (602);a type of sensor of the first XR device (602); andan interaction mechanism of the first XR device (602).

12. The method as claimed in any of the preceding claims, wherein the output comprises one or more of:a visual output;a haptic output; andan audio output.

13. The method as claimed in any of the preceding claims, wherein the original program is configured to be executed to generate a display at the first XR device (602).

14. The method as claimed in any of the preceding claims, wherein the first XR device (602) is one or more of:an augmented reality, AR, device;a virtual reality, VR, device; anda mixed reality, MR, device.

15. The method as claimed in any of the preceding claims, wherein the first XR device (602) comprises one or more of:a user equipment, UE;a wearable device;a headset;glasses;a television;a smartphone; anda tablet.

16. The method as claimed in any of the preceding claims, wherein the original program is a device agnostic program.

17. The method as claimed in any of the preceding claims, wherein providing (310, 636) the modified program comprises:providing the modified program to the runtime system (608) instead of the original program.

18. The method as claimed in any of the preceding claims, wherein the original program is created using a second API layer.

19. The method as claimed in claim 18, wherein both the first API layer (606) and the second API layer are associated with a set of corresponding API layers.

20. The method as claimed in any of the preceding claims, wherein the original program comprises at least one information element, and wherein the at least one information element is indicative of a requirement to modify the original program.

21. The method as claimed in claim 20, wherein modifying (308, 634) the original program comprises:modifying the original program in response to identifying the at least one information element.

22. The method as claimed in any of the preceding claims, wherein modifying (308, 634) the original program comprises modifying at least a portion of code of the original program.

23. A computer-implemented method for handling a program for an extended reality, XR, device, the method comprising:providing (402, 618), by a first XR device (602), a first request for initiation of a runtime system (608);providing (404, 630), by the first XR device (602), a second request for the runtime system (608) to execute an original program, wherein the original program is configured to be executed to generate an output at the first XR device (602);in response to providing (404, 630) the second request:obtaining (406, 636), by the first XR device (602), one or more instructions from the runtime system (608), wherein the one or more instructions are generated by execution of a modified program, wherein the modified program corresponds to a modified version of the original program, and wherein the original program is modified, by a first application programming interface, API, layer (606), based on first information indicative of one or more characteristics of the first XR device (602); andexecuting (408), by the first XR device (602), the one or more instructions.

24. The method as claimed in claim 23, wherein executing (408) the one or more instructions comprises:generating the output, by the first XR device (602), based on the one or more instructions.

25. The method as claimed in claim 24, wherein:the first request is provided to the first API layer (606); and / orthe second request is provided to the first API layer (606).

26. The method as claimed in claim 23 or 25, wherein providing (402, 618) the first request comprises providing a system call comprising the first request.

27. The method as claimed in any of claims 23 to 26, wherein:the first information is stored in a memory of the first XR device (602); and / or the original program is stored in a memory of the first XR device (602).

28. The method as claimed in claim 27, the method comprising:obtaining, by the first XR device (602), the original program from the memory of the first XR device (602).

29. The method as claimed in any of claims 23 to 28, wherein the runtime system (608) is at least partially comprised in:the first XR device (602); and / ora cloud network.

30. The method as claimed in any of claims 23 to 29, wherein the one or more characteristics of the first XR device (602) comprise:one or more hardware characteristics of the first XR device (602); and / or one or more functional characteristics of the first XR device (602).

31. The method as claimed in any of claims 23 to 30, wherein the one or more characteristics of the first XR device (602) comprise one or more of:a type of the first XR device (602);a form factor of the first XR device (602);a type of a display of the first XR device (602);a size of the display of the first XR device (602);a display resolution of the display of the first XR device (602);a type of sensor of the first XR device (602); andan interaction mechanism of the first XR device (602).

32. The method as claimed in any of claims 23 to 31 , wherein the output comprises one or more of:a visual output;a haptic output; andan audio output.

33. The method as claimed in any of claims 23 to 32, wherein the original program is configured to be executed to generate a display at the first XR device (602).

34. The method as claimed in any of claims 23 to 33, wherein the first XR device (602) is one or more of:an augmented reality, AR, device;a virtual reality, VR, device; anda mixed reality, MR, device.

35. The method as claimed in any of claims 23 to 34, wherein the first XR device (602) comprises one or more of:a user equipment, UE;a wearable device;a headset;glasses;a television;a smartphone; anda tablet.

36. The method as claimed in any of claims 23 to 35, wherein the original program is a device agnostic program.

37. The method as claimed in any of claims 23 to 36, wherein the original program is created using a second API layer.

38. The method as claimed in claim 37, wherein both the first API layer (606) and the second API layer are associated with a set of corresponding API layers.

39. The method as claimed in any of claims 23 to 38, wherein the original program comprises at least one information element, and wherein the at least one information element is indicative of a requirement to modify the original program.

40. A method performed by a system, the method comprising:the method as claimed in any of claims 1 to 22; and / orthe method as claimed in any of claims 23 to 39.

41. A first entity (10) comprising:processing circuitry (12) configured to cause the first entity (10) to:obtain, by a first application programming interface, API, layer (606), a first request for initiation of an runtime system (608), wherein the first request is requested by a first XR device (602);obtain, by the first API layer (606), first information indicative of one or more characteristics of the first XR device (602);obtain, by the first API layer (606), a second request for the runtime system (608) to execute an original program, wherein the original program is configured to be executed to generate an output at the first XR device (602);in response to obtaining the second request:modifying, by the first API layer (606), the original program based on the first information to generate a first modified program; and providing, by the first API layer (606), the first modified program to the runtime system (608) for execution of the modified program.

42. A first entity (10) as claimed in claim 41 , wherein:the processing circuitry (12) is configured to cause the first entity (10) to perform the method of any of claims 2 to 22.

43. A first entity (10) as claimed in claim 41 or 42, wherein the first entity (10) comprises the first XR device.

44. A second entity (20) comprising:processing circuitry (22) configured to cause the second entity (20) to:provide, by a first XR device (602), a first request for initiation of a runtime system (608);provide, by the first XR device (602), a second request for the runtime system (608) to execute an original program, wherein the original program is configured to be executed to generate an output at the first XR device (602); and in response to providing (630) the second request:obtain, by the first XR device (602), one or more instructions from the runtime system (608), wherein the one or more instructions are generated by execution of a modified program, wherein the modified program corresponds to a modified version of the original program, and wherein the original program is modified, by a first application programming interface, API, layer (606), based on first information indicative of one or more characteristics of the first XR device (602); andexecute, by the first XR device (602), the one or more instructions.

45. A second entity (20) as claimed in claim 44, wherein:the processing circuitry (22) is configured to cause the second entity (20) to perform the method of any of claims 24 to 39.

46. A second entity (20) as claimed in claim 44 or 45, wherein the second entity (20) comprises the first XR device.

47. A system comprising:at least one first entity (10) as claimed in any of claims 41 to 43; andat least one second entity (20) as claimed in any of claims 44 to 46.

48. A computer program comprising instructions which, when executed by processing circuitry, cause the processing circuitry to perform the method according to any of claims 1 to 22, and / or the method according to any of claims 23 to 39.

49. A computer program product, embodied on a non-transitory machine-readable medium, comprising instructions which are executable by processing circuitry to cause the processing circuitry to perform the method according to any of claims 1 to 22, and / or the method according to any of claims 23 to 39.

Images