Mapping states of a physical object interacted with by a user to provide operation command inputs to an extended reality application

Abstract

Operations by an XR device detect a physical object, OBJ, in a field of view, FOV, of an image sensor. Operations determine a number of states of the OBJ that satisfies a rule for a user to interact with the OBJ to provide operational command inputs to an application. Operations determine whether the number of determined states is less than a number of operation commands compatible as inputs to a current operational state of the application. Responsive to determining that the number of determined states is less than the number of operation commands of the application, operations select a subset of the operation commands of the application. Operations assign each operation command of the subset to a different state of the OBJ.

Description

MAPPING STATES OF A PHYSICAL OBJECT INTERACTED WITH BY A USER TO PROVIDE OPERATION COMMAND INPUTS TO AN EXTENDED REALITY APPLICATIONTECHNICAL FIELD

[0001] The present disclosure relates to an extended reality (XR) device for providing an improved extended reality interface which allows for using physical objects in the surroundings as input objects.BACKGROUND

[0002] Augmented reality (AR) is a technology that overlays digital information or computergenerated graphics onto the physical world. It involves adding virtual elements to the user's actual environment, typically viewed through a camera (Video see-through (VST) or video pass-through) or a transparent screen (Optical see-through XR), such as in a headset, a full-body suit or through a smartphone. AR can enhance the perception of reality by adding layers of digital information that can be informative, entertaining, or both, or neither.

[0003] Virtual reality (VR), on the other hand, creates a completely immersive, computer-generated environment that can be experienced through a headset or a full-body suit. In VR, users are visually isolated from the real world and transported to a simulated environment that can be interactive and often responsive to user input.

[0004] The key difference between AR and VR is that AR enhances the physical world by overlaying digital elements onto it, whereas VR replaces the physical world with a computer-generated environment. AR appears therefore to be a more natural and intuitive experience that blends the virtual with the real world, while VR is a more immersive but completely synthetic experience.

[0005] Both AR and VR have a wide range of applications, including gaming, education, advertising, training, and simulation, among others.

[0006] Mixed reality (MR) is a term used to describe the merging of a real-world environment and a computer-generated one. Physical and virtual objects may co-exist in mixed reality environments and interact in real time.

[0007] Extended reality (XR) is a catch-all term to refer to augmented reality (AR), virtual reality (VR), and mixed reality (MR).

[0008] Head-worn devices, such as optical-see-through, XR, devices, for example XR glasses, are used more and more in AR and XR systems, and are thus becoming more and more commonplace.

[0009] When navigating an extended reality environment, such as in an augmented reality, the user may use specific gloves (or other devices) to give input or other control to the system. Such devices require that the user brings them along and that they are kept ready at hand to be used. Alternatively, the user may use gestures to give input or other control. Such gestures may be difficult to remember, especially since they may differ between applications and also, if there are many applications having different commands or options, the resulting number of gestures a user has to learn becomes quite high.

[0010] The patent document published as US2022101002A1 discloses a method of generating a virtual reality (" VR") experience includes recognizing a real-world (" RW") object in possession of a user of the VR experience and defining a VR object that corresponds to the RW object, wherein the VR object is configured to be used during the VR experience. The method also includes tracking the RW object to identify actions related to with the RW object and rendering the VR object in the VR experience to correspond with the actions related to the RW object. This document discusses a highly advanced system based on artificial intelligence learning how a user handles real-world objects. One example is where a user is able to handle a smartphone also in the virtual reality and where for example an image taken in the virtual reality is stored in the smartphone in the real-world.

[0011] The patent document published as US2021031110A1 discloses a system for generating video game inputs is provided. The system comprises an input unit operable to obtain images of a passive non-luminous object being held by a user as a video games controller. The system also comprises a physical object detector and object pose detector for detecting the physical object and its respective pose in the obtained images. The pose detector is configured to detect the pose of the physical object based on at least one of a (i) contour detection operation and (ii) the output of a machine learning model that has been trained to detect the poses of passive non-luminous objects in images. A user input generator is configured to generate user inputs based on the detected changes in pose of the passive non-luminous object and to transmit these to a video game unit at which a video game is being executed. A corresponding method is also provided. In this document any object picked up by the user may be used as a game controller and is thus restricted to objects already picked up and selected by the user.SUMMARY

[0012] Various embodiments of the present disclosure are directed to improving how users can interact with physical objects in their environment to control operation of an XR device including instances when the physical object(s) includes a number of states that are less than a number of required operation commands of an application.

[0013] In some embodiments, an XR device includes a controller. The controller is operative to detect a physical object, OBJ, in a field of view, FOV, of an image sensor, determine a number of states of the OBJ that satisfies a rule for a user to interact with the OBJ to provide operational command inputs to an application, and determine whether the number of determined states is less than a number of operation commands compatible as inputs to a current operational state of the application. The controller is further operative to, responsive to determining that the number of determined states is less than the number of operation commands of the application, select a subset of the operation commands of the application. The controller further is operative to assign each operation command of the subset to a different state of the OBJ.

[0014] Potential advantages provided by various embodiments of the present disclosure includes an XR device adapting an XR application to a physical reality present for a user by enabling mapping / re-mapping of in-application controllers to physical objects that would otherwise be considered not fit for purpose or not useable. Thus, enabling a user to interact with the application despite the real-world physical environment's "available useful objects"-shortcomings.

[0015] Other XR devices and corresponding methods and computer program products according to embodiments of the present disclosure will be or become apparent to one with skill in the art upon review of the following drawings and detailed description. Moreover, it is intended that all embodiments disclosed herein can be implemented separately or combined in any way and / or combination.BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Embodiments of the invention will be described in the following, reference being made to the appended drawings which illustrate non-limiting examples of how the inventive concept can be reduced into practice.

[0017] Figure 1A shows a schematic view of electronic components that can be integrated into or operationally connected to an XR device according to some embodiments of the present disclosure,

[0018] Figure IB shows a schematic view of an XR device according to some embodiments of the present disclosure,

[0019] Figures 2A, 2B and 2C each shows a schematic view of a physical object and possible stable mechanical equilibrium states for such a physical object,

[0020] Figures 3A to 3H each shows a schematic view of an XR device according to some embodiments of the present disclosure,

[0021] Figures 4A to 4B each shows a schematic view of an XR device according to some embodiments of the present disclosure,

[0022] Figure 5 shows a flowchart of operations by the XR device according to some embodiments of the present disclosure,

[0023] Figure 6 shows a component view for a software component arrangement according to some embodiments of the present disclosure,

[0024] Figure 7 shows a flowchart of operations by the XR device for handling three scenarios relating to comparison of a number of user input interfaces and a number of stable equilibrium states of an available physical object(s) that are to be interacted with by a user to provide input to an operating system program and / or an application program of the XR device, according to some embodiments of the present disclosure,

[0025] Figure 8A shows an example of an in-application texture for an in-application controller that includes eight operation commands, according to some embodiments of the present disclosure,

[0026] Figure 8B shows another example of an in-application texture for an in-application controller that includes three operation commands, according to some embodiments of the present disclosure, and

[0027] Figure 9 shows a flowchart of operations by the XR device for mapping states of a physical object interacted with by a user to provide operation command inputs to an XR application, according to some embodiments of the present disclosure.DETAILED DESCRIPTION

[0028] Inventive concepts will now be described more fully hereinafter with reference to the accompanying drawings, in which examples of embodiments of inventive concepts are shown. Inventive concepts may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure willbe thorough and complete, and will fully convey the scope of various present inventive concepts to those skilled in the art. It should also be noted that these embodiments are not mutually exclusive. Components from one embodiment may be tacitly assumed to be present or used in another embodiment.

[0029] Figure 1A shows a schematic view of electronic components that can be integrated into or operationally connected to an XR device 100 according to some embodiments of the present disclosure. Hereafter both will be referred to as the XR device 100. The XR device 100 comprises a controller 101 and optionally a memory 102. The XR device 100 may comprise a XR head mounted device (HMD), a handheld XR device, or a networked computer server networked to a XR HMD or networked to a handheld XR device.

[0030] The XR device 100 may also comprise or be connected to an image sensor 103. The XR device 100 may further comprise or be connected to a display device 104.

[0031] The controller 101 is configured to control the overall operation of the XR device 100. In some embodiments, the controller 101 may be a general-purpose controller, e.g., general purpose processor, which can perform a variety of tasks. As a skilled person would understand there are many alternatives for how to implement a controller, such as using Field -Programmable Gate Arrays (FPGA) circuits, application-specific integrated circuits (ASICs), graphics processing units (GPUs), etc. in addition to or as an alternative. For the purpose of this application, all such possibilities and alternatives will be referred to simply as the controller 101.

[0032] It is also noted that in some embodiments, parts of or all of the processing of the controller may be performed remotely, where a local controller 101 is configured to provide input data to a remote processing platform, such as to a cloud server or an edge network node, causing the remote processing platform to perform the processing and transmit the results of such processing as output to the local controller 101. For the purpose of this application, such possibilities and alternatives will also be referred to simply as the controller 101. The controller 101 may thus represent both the local controller and the remote processing platform.

[0033] The memory 102 is configured to store map data, graphics data, User Interface (Ul) settings and computer-readable instructions (e.g., operating system instructions, application program instructions, etc.) that when loaded into the controller 101 indicate how the XR device 100 is to be controlled. The memory 102 may comprise one or more discrete memory circuits or hierarchy of memories, which are collectively referred to as being part of the same overall memory 102. There maybe one memory used by the display device storing graphics data, another memory used by the sensor for storing settings, another memory used by the communications interface (if such is present) for storing settings, and so on. As a skilled person would understand, there are many possibilities of how to select where data should be stored. In one embodiment, the XR device may comprise a general memory 102 in turn comprising any and all such memory circuits for the purpose of this application. As a skilled person would understand there are many alternatives of how to implement a memory, for example using non-volatile memory circuits, such as EEPROM memory circuits, or using volatile memory circuits, such as RAM memory circuits. For the purpose of this application all such alternatives will be referred to simply as the memory 102.

[0034] The display device 104 may in some embodiments be a display arrangement comprising one or more displays arranged to present visual data, predominantly through images. In some such embodiments, the display device 104 may be a touch screen thereby enabling user input to be provided to and received by the XR device 100. The visual data is related to the user interface of the XR device being presented by the XR viewing device 100. The XR device is thereby arranged to present image data through a (graphical) user interface in a manner controlled by the controller 101. When implemented as a HMD XR device, the display device 104 may be configured as an optical see-through (OST) screen on which the user views video of the real-world captured by the image sensor 103, or may be configured as a see-through screen which allows the user to view the light from the real-world passed through the display screen with overlaid computer-generated graphics providing an augmented reality experience.

[0035] The image sensor 103, such as a camera or image sensor module, is arranged to provide an image (or stream of images) of the user environment when the user is utilizing the XR device 100, wherein images of the user environment may be analyzed using image processing techniques known in the art in order to determine or detect one or several objects in the surroundings. As a skilled person would understand, the XR device 100 may comprise one controller 101 and the sensor 103 may comprise another controller, but for the purpose of the teachings herein, they will be considered to be the same controller 101 in order to cover all possible variations of exactly where the determination of a physical object takes place.

[0036] In some embodiments, the image sensor 103 alternatively comprises non-visible light scanning including light detection and ranging (LiDAR) that provides information of the user environment when the user is utilizing the XR device 100, wherein the LiDAR information of the userenvironment may be analyzed using processing techniques known in the art in order to determine or detect one or several objects in the surroundings.

[0037] Figure 1B shows a schematic illustration of an XR device 100 in the form of a head-worn XR device in which the image sensor 103 provides a video stream capturing the real-world to the controller 101 for display on the display device 104. The controller 101 can operate to detect a physical object in the video stream from the image sensor 103.

[0038] The XR device 100 may in some embodiments be at least partially implemented in a smartphone. The smartphone may be carried in a head-worn carrier that enables the smartphone to be used as an XR headset while the head-worn carrier is worn on the head of the user.

[0039] In some embodiments the XR device 100 is an AR or XR headset (also referred to as goggles).

[0040] As mentioned above, the inventors have realized that by determining states of objects in the surroundings (as perceived by the image sensor 103) and assigning operation commands to these states, objects that the user perceives may be interacted with by the user to provide input (e.g., program control) for giving operation commands (also referred herein as commands) to the XR device 100, e.g., as user input provided to an operating system (OS) program and / or to an application program executing in the XR environment.

[0041] In some embodiments, the states may include a stable mechanical equilibrium state or another type of state that satisfies a rule for a user to interact with an object to provide operational command inputs to an application. The rule(s) may include different characteristics (affordances) of an object that would allow the user to provide operation command inputs to an application.

[0042] The affordances of an object may include (without limitation) a surface's macro / microstructure of the physical object (e.g., surface size or curvature), type of material of the physical object (e.g., exterior and / or internal material), edge of the physical object (e.g., edge length or curvature), moveable structure of the physical object (e.g., clicker on a pen), texture of the physical object (soft, hard, rough, etc.), feedback produced by the physical object (e.g., tactile auditory), elasticity of the object (e.g., elastic, bendable, rigid), size of the physical object -- a user's interaction with the physical object (e.g., how grabbable is the object), temperature of the physical object, etc. For example, a click-pen with a clicker on the end of the pen may include a state that satisfies a rule through the use of the clicker on the pen (that would allow the pen to push out / in the ink cartridge in the real world) that would allow the user to click the pen to provide operational command inputs (e.g., on / off input) to an application. The click-pen may include additional states that satisfies a different or same rule if theclick pen includes other affordances that satisfy the rule or if the pen includes a mechanical equilibrium state.

[0043] A stable mechanical equilibrium state as referred to herein is to be understood to be a mechanical equilibrium state. In classical mechanics, a particle is in mechanical equilibrium if the net force on that particle is zero. By extension, a physical system made up of many parts is in mechanical equilibrium if the net force on each of its individual parts is zero. A stable mechanical equilibrium state includes a state where the physical object will come to rest, unless subjected to external forces. While stable mechanical equilibrium states are primarily discussed herein, other types of states may also be similarly determined and used herein.

[0044] The inventors have realized that in most cases, a user's physical environment may not have many regular-shaped objects that are easy to interact with, but that even irregularly shaped objects may be used, by determining a encompassing shape of the physical object.

[0045] The controller 101 can operate to determine the states of an object based on detecting and recognizing a physical object, or detecting and recognizing affordances of the object. For example, for stable mechanical equilibrium states, the controller 101 can operate to determine the stable mechanical equilibrium states based on detecting and recognizing a physical object and then determining an encompassing shape of the physical object. Then the stable mechanical equilibrium states of the encompassing shape are determined and used as the stable mechanical equilibrium states of the physical objects. Figure 2A shows an example of a physical object (OBJ) having an irregular shape, in this example two triangles joined at their tops or edges. Figure 2A also shows an encompassing shape 105 having been determined for the physical object.

[0046] In some embodiments, the stable mechanical equilibrium states of a physical object (or the stable mechanical equilibrium states of the encompassing shape for a physical object) are determined as being the states, when the physical object (or encompassing shape thereof) is resting on an equilibrium plane. An equilibrium plane of a physical object - i.e., a plane on which the physical object is laying under gravity - is defined by the normal plane direction n. In order to find the equilibrium planes of a physical object (such as exemplified by a 3D-model), the (convex-)encompassing shape and the center of mass of the physical object first need to be computed.

[0047] It is noted that in the description below, reference is made to a 3D model of a physical object. However, these references should be assumed to be also to the physical object as detected bythe imaging sensor. The 3D model may thus also be referenced as a 2D model, being a specific nonlimiting example of a 3D model.

[0048] The convex encompassing shape (or hull) of a physical object is the smallest convex polyhedron containing the physical object the physical object. It is a fundamental construction for mathematics and computational geometry and will therefore not be discussed in great detail as a skilled person can find details on this through various known articles and textbooks. Its construction can be achieved in many ways. Some operations that may be used by the controller 101 are based on deterministic incremental algorithms or randomized incremental construction. In some embodiments a CGAL implementation of the QuickHull algorithm is utilized. It handles all degenerate cases and nonmanifold models of objects that constitute 3D-model database. In the worst case this algorithm is O(n2), but in practice it is no worse than O(n*log(n)), where n is the number of vertices of a 3D-model.

[0049] The calculation of the center of mass and the volume of rigid bodies has also been extensively treated in literature. One efficient approach that can be used by the controller 101 to compute the center of mass for polyhedral objects will be discussed, but it should not be that there are many other alternative algorithms available. This algorithm is based upon a three-step reduction of the volume integrals to successively simpler integrals. The final step of the algorithm computes the required integrals over a face from the coordinates of the projected vertices. Considering that mass distribution is homogeneous in the 3D-model volume V, the center of mass G thus yields:

[0050] An orthogonal projection along the z direction thus results in a more convenient and efficient integral, as follows:

[0051] where S is the surface boundary of the 3D-model, n is the normal at point P(x, y, z) and sign(x) denotes the signum function which extracts the sign of a real number x.

[0052] Using the divergence theorem, the 3D-model volume V yields:

[0053] The overall complexity of the two integrals is O(m), where m represents the number of faces of the polyhedron.

[0054] Based on this the equilibrium planes may be computed by assuming the following conditions to compute an equilibrium plane:

[0055] Theorem 1 A direction n defines an equilibrium plane E if and only if there exists a plane n of normal n such that it contains a face Fi of the convex encompassing shape H of the 3D-model / object (convex encompassing shape condition) and if the projection of the center of mass G of the 3D-model / object along n is inside Fi (center of mass condition).

[0056] Proof A static mechanical equilibrium requires that at least three non collinear points of the 3D-model belong to n. Since n is not a separating plane of the 3D-model points, it necessarily contains a face Fi of the convex encompassing shape H. Let us suppose the projection of G along n onto n does not belong to Fi. The gravity force applied on G thus exerts a moment whose axis A is the nearest edge of Fi from G. The ground force also exerts a moment on A, that does not oppose to the gravity moment. So their resultant moment on A is not zero and the 3D-model is thus not in static equilibrium. The projection of G along n onto n must consequently belong to Fi.This provides the equilibrium plane computation:Algorithm 1 Equilibrium plane algorithmCompute the convex hull.Compute the center of mass G.for all faces Fi doProject orthogonally G on plane π containing Fi, / M;if G' ∈ Fi thenAdd direction n to the equilibrium plane listend ifend for

[0057] The Computational Geometry Algorithms Library, CGAL, comprises many algorithms that are available to a skilled person for implementing such determinations as herein. The CGAL is a software project that provides easy access to efficient and reliable geometric algorithms in the form of a C++ library.

[0058] As noted above, stable mechanical equilibrium states may thus be determined for any object in the user's environment and these stable mechanical equilibrium states may be assignedcommands. A user may thus be enabled to input commands or other control by interacting with these objects in the surroundings.

[0059] Figure 2B shows a schematic illustration of how a encompassing shape 105 can be placed in different positions or be placed in different stable mechanical equilibrium states. The encompassing shape 105 is the same as that of Figure 2A, and it can assume three stable mechanical equilibrium states (in two dimensions). As mentioned above, the teachings herein are discussed with relation to two dimensions, but they equally apply to three dimensions. In Figure 2B three states are shown; standing, laying and standing up-side-down. Depending on the shape of the physical object, there may be different number of states for the laying state in Figure 2B. Figure 2C shows examples of the number of different states depending on the shape of the encompassing shape 105 of the physical object in Figure 2A. For illustrative purposes, four different examples are shown where the encompassing shape has 3, 4, 5 or 6 different stable mechanical equilibrium states (when laying).

[0060] The general operation of an XR device 100 according to some embodiments, and specifically as illustrated in Figures 1A and / or 1B, will now be discussed with reference to Figures 3A to 3H. In the following discussion operations are described in the context of a user interacting with a physical object to provide user input to the OS and / or application program operating in the XR device 100. The user input is also referred to as a command that input by the user, whereby different user input can correspond to different commands being input by the user. The term "command" below can therefore be replaced with the more general term "user input" or "input".

[0061] Figure 3A shows a schematic view of an XR device 100 that detects a physical object in the surroundings utilizing the image sensor 103 and determines an encompassing shape 105 for the physical object (Figure 3B) and the stable mechanical equilibrium states for the encompassing shape 105. The XR device 100 also assigns commands to at least some of the states of the encompassing shape 105 (after determining that the physical object is a usable physical object). In order to enable the user to interact with the physical object correctly, the XR device 100 may, in some embodiments, display the assigned commands on the display device 104, as virtual labels or other graphic indicators 104-R, Figure 3C.

[0062] The XR device then detects an interaction with of the usable physical object, Figure 3D. The interaction may be detected by tracking the hands of the user, or simply by detecting a change in the placement of the usable physical object. As the usable physical object has been detected to have been interacted with the XR device 100 determines which stable mechanical equilibrium state that the usablephysical object (or the encompassing shape thereof) has assumed or been placed in, and then executes the command that is assigned to that stable mechanical equilibrium state, Figure 3E.

[0063] The teachings herein thus provide an XR device 100 configured to, detect a usable physical object in the field of view (FOV) of the user (i.e. the surroundings as perceived through video output by the image sensor 103). In some embodiments the physical object is defined as a physical object having two or more stable mechanical equilibrium states, and the XR device 100 is configured to determine two or more stable mechanical equilibrium states of the physical object, assign the one or more operation commands of the application to the stable mechanical equilibrium states of the physical object, detect a user interaction with the physical object, determine a resulting stable mechanical equilibrium state of the usable physical object after user interaction, and execute the operation command assigned to the resulting stable mechanical equilibrium state.

[0064] It is noted that the example of Figures 3A to 3E show the usable physical object being interacted with from a first placement / state to a second state. However, the association of commands may also be made for a series of interactions where the command is not only associated to the resting or final state, but to a defined series of intermediate state(s). A first command may thus be associated with placing the usable physical object directly in a first stable mechanical equilibrium state (for example turning up-side-down). A second command may thus be associated with placing the usable physical object in the second stable mechanical equilibrium state (up-side-down), but by first placing it in an intermediate state (laying). The teachings herein thus provide, in some embodiments, an XR device 100 further configured to determine one or more intermediate stable mechanical equilibrium states assumed during the user interaction with the usable physical object, and wherein the command assigned to resulting stable mechanical equilibrium state depends on the one or more intermediate stable mechanical equilibrium states assumed during the user interaction.

[0065] As the inventors have realized, not all physical objects are usable for user interaction. Thus, some teachings herein provide for detecting only usable physical objects. However, some teachings further below, provide for detecting physical objects that are not considered usable physical objects (e.g., objects that include a number of states that is less than a number of operation commands compatible as inputs to a current operational state of an application).

[0066] The teachings that provide for detecting only usable physical objects are discussed first below.

[0067] A physical object may be usable if it has enough states to represent the commands of the current XR environment / application. If a physical object only has two states, it may be deemed to not be usable for an application having five commands. The teachings herein thus provide, in some embodiments, an XR device 100 further configured to determine a required number of operating commands of the application and to detect a physical object as a usable physical object if the number of stable mechanical equilibrium states of the physical object is equal to or higher than the required number of operating commands.

[0068] If a physical object is deemed to not be suitable, the XR device 100 is determined, in some embodiments, to detect a second physical object and determine if the second object may be a usable object, Figure 3F. The teachings herein thus provide, in some embodiments, an XR device 100 further configured to determine another physical object if the number of stable mechanical equilibrium states of the first object is less than the required number of command operations.

[0069] As the inventors have realized, a physical object having more stable mechanical equilibrium states than the commands of the XR environment / application may still be utilized, and a command may be associated with more than one stable mechanical equilibrium state. The teachings herein thus provide, in some embodiments, an XR device 100 further configured to assign at least one of the operating commands to multiple stable mechanical equilibrium states when assigning operating commands to stable mechanical equilibrium states, when the number of stable mechanical equilibrium states is higher than the number of operating commands.

[0070] Specifically, the teachings herein thus provide, in some embodiments, an XR device 100 further configured to detect a physical object as a useful physical object if the number of stable mechanical equilibrium states is equal to the number of operating commands.

[0071] As the inventors have also realized, a physical object may still be usable - even if the number of stable mechanical equilibrium states is less than the number of commands - by assigning commands to sequences of stable mechanical equilibrium states (as discussed above with reference to intermediate stable mechanical equilibrium states).

[0072] The inventors have also realized that even if an encompassing shape or shape of a physical object may have many stable mechanical equilibrium states, not all these states are possible for a physical object. For example, a cup may not be possible to place in all states of the corresponding encompassing shape, such as up-side-down. The teachings herein thus provide, in some embodiments, an XR device 100 is further configured to determine a number of possible stable mechanical equilibriumstates that the possible object can assume and set the number of possible states that can be assumed as the number of stable mechanical equilibrium states for the physical object.

[0073] In some embodiments the XR device is configured with (such as being stored in the memory 102 or being retrievable from a networked computer server) a library of stored physical objects and associated possible encompassing shapes. A physical object that is detected can be compared to the library of encompassing shapes in order to determine the number of possible stable mechanical equilibrium states. The teachings herein thus provide, in some embodiments, an XR device 100 that is further configured to determine a number of possible stable mechanical equilibrium states that the possible object can assume by matching the encompassing shape to stored shapes, wherein the stored shapes have unallowable stable mechanical equilibrium states.

[0074] Alternatively or additionally, the detected physical object may be classified or otherwise identified through a physical object segmentation. Such object segmentations are known in the art of image processing and will not be further discussed herein. The physical object segmentation may thus identify a physical object as a mug, which has an unallowable state being up-side-down associated with it. The teachings herein thus provide, in some embodiments, an XR device 100 further configured to determine a number of possible stable mechanical equilibrium states that the possible physical object can assume by utilizing object segmentation to identify a type of physical object, wherein some types of physical objects have unallowable stable mechanical equilibrium states.

[0075] Alternatively or additionally, the XR device may further incorporate a thermal imaging sensor (assumed to be part of the imaging sensor 103), whereby a physical object may be deemed unusable based on a temperature of the physical object. For example, a physical object that has a high temperature may be determined to be unsuitable for user interaction. Similarly, a physical object may be determined as unsuitable if it is too cold. The teachings herein thus provide, in some embodiments, an XR device 100 further configured to determine a number of possible stable mechanical equilibrium states that the possible physical object can assume by determining that the physical object has a temperature that is above, below or at a temperature threshold level, and wherein the controller is configured to in that case set the number of possible stable mechanical equilibrium states for the possible object to zero.

[0076] Alternatively or additionally, the XR device may further incorporate an inertial sensor (assumed to be part of the sensor 103), whereby the XR device can determine that it is in a moving (or shaking) environment such as on a bus or a train, wherein a number of states may not be possible for aencompassing shape. Such as when a state requires only small forces to be toppled. The teachings herein thus provide, in some embodiments, an XR device 100 further configured to determine a number of possible stable mechanical equilibrium states that the possible physical object can assume by determining that the inertial sensor (IMU) 103 indicates a movement level that exceeds a movement threshold level, and wherein the controller is configured to in that case set the number of possible stable mechanical equilibrium states for the possible physical object to zero. For example, the physical object (e.g., whiteboard pen) may always be stable within reasonable forces such as being expected while a user interacts with the user environment (e.g., office table environment). In this example, a user may bump into a table, that the physical object is resting on, without making physical object fall over. But the physical object may not be "stable" if put on a dashboard of a moving car since the perturbation forces (e.g., from bumps picked up from road) in that context become too large.

[0077] If the XR device is not able to detect any usable physical object in the current environment (field of view), the XR device 100 may be configured in some embodiments to prompt the user to change the environment. In some embodiments the environment may be changed by moving (laterally and / or rotationally) so that a new environment moves into the field of view, whereby new usable physical objects will come into view. In some embodiments the environment may be changed by moving physical objects around in the current view, whereby new objects may come into view, such when certain physical objects not usable for any action in the XR environment or XR applications partially or entirely obscure usable physical objects. Figures 3G and 3H show an example where two objects may be identified as a single object (Figure 3G) not having many usable states, but when moved around (figure 3H) actually reveals two objects each having usable states. The teachings herein thus provide, in some embodiments, an XR device 100 further configured to indicate to the user to change environment when no usable physical object can be detected.

[0078] Alternatively or additionally, the XR device is configured to enable a user to select which usable physical object to be used for interaction with. In some such embodiments the XR device is configured to indicate usable physical objects to a user, by for example marking them on the display 104, and then receiving an indication from the user as to which usable physical object should have motion and / or location tracked for providing user input. For example, the user may be presented with graphical markings indicating two usable physical objects, and then select one of them by pointing at the usable physical object selected. The imaging sensor 103 will be able to track the hand and detect the pointing, whereby the pointed to usable physical object will be identified by the controller 101. Theteachings herein thus provide, in some embodiments, an XR device 100 further configured to select the usable physical object by indicating one or more usable physical objects to the user, receiving user indication of one usable physical object and selecting the indicated usable physical object for controlling the XR environment or the XR application currently running.

[0079] As discussed in the above, the commands associated with a usable physical object may be indicated graphically to the user. As is shown in Figures 4A and 4B, the commands associated with a usable physical object may be indicated through a graphical representation 104-R on the display 104, wherein a graphical indication 105R of the usable physical object may be shown, such as a graphical indication of the encompassing shape of the usable physical object. A label or other graphical indication of a command 104C may be shown on or adjacent to the side of a state associated with the command. In Figure 4A there are two commands C0M1, COM2 indicated through graphical indications 104C1, 104C2 along the associated sides.

[0080] As a usable physical object is interacted with the XR device may be arranged to indicate that the associated command is being (or to be) executed. This enables a user to understand the consequence of the interaction. In some embodiments (general) confirmation is required to execute the command. The indication is in some embodiments provided by changing or adapting the graphical indication of the command. In Figure 4B this is shown as the graphical indication 104C2 for the second command COM2 having been changed as the physical object has been interacted with (placed in a laying position).

[0081] The teachings herein thus provide, in some embodiments, an XR device 100 further configured to indicate to the user at least one assigned operating command to a stable mechanical equilibrium state of the encompassing shape.

[0082] As noted above, the stable mechanical equilibrium states for a usable physical object are determined based on the encompassing shape of the usable physical object in the form of a (convex) hull. The XR device 100 is thus, in some embodiments, configured to determine an encompassing shape for the physical object by determining a convex hull encompassing the usable physical object.

[0083] However, other manners of determining a shape encompassing the usable physical object may be used, and the teachings herein relating to a hull, thus also apply to any other shape used. In some embodiments the XR device 100 is further configured to determine an encompassing shape for the usable physical object by determining a shape for the detected usable physical object and matching the shape of the detected usable physical object to known shapes stored in the memory 102.

[0084] Figure 5 shows a general flowchart for various operations by the XR device 100 as discussed above.

[0085] Although operations herein are illustrated in flowcharts as being performed in a sequence, it is to be understood that the order of operations can be changed without restriction, that two or more operations that are shown in sequence may be performed in parallel, and that two or more of the operations may be performed repetitively more often than other illustrated operations.

[0086] The operations of Figure 5 include determining 510 one or more operating commands or other user input interfaces applicable to an XR environment or an application running in the XR environment, detecting 520 a usable physical object in the field of view of the user, determining 530 an encompassing shape for the usable physical object, wherein the encompassing shape is a geometrical shape that encompasses the usable physical object. The operations further include determining 540 two or more stable mechanical equilibrium states of the encompassing shape, selecting 550 the usable physical object as a command object (or other user input interface object) to be used. The command (user interface object) object can be a usable physical object seen by the user through the XR device and when interacted with by the user indicates a command (input to a user interface) to be performed by the XR device, and assigning 560 each of the one or more operating commands (user input interfaces) of the application to each stable mechanical equilibrium state of the encompassing shape for the command object. The operations further include detecting 570 a user interaction of the command object, determining 580 a resulting stable mechanical equilibrium state of the command object after user interaction and executing 590 the command (user input interface) assigned to the resulting stable mechanical equilibrium state.

[0087] Figure 6 shows a component view for a software component or module arrangement 600 according to some embodiments of the teachings herein. The software component arrangement 600 is adapted to be used in an XR device 100 as taught herein for providing adaptation as taught herein and corresponds to the operation of the XR device 100 in the above. The software component arrangement 600 comprises a software component 610 for a software component for determining one or more operating commands (user input interfaces) that can be into to an OS program of an XR environment or an application program running in the XR environment, a software component 620 for detecting a usable physical object in the field of view of the user, a software component for determining 630 an encompassing shape for the usable physical object, wherein the encompassing shape is a geometrical shape that encompasses the usable physical object. The software component arrangement 600 furthercomprises a software component 640 for determining two or more stable mechanical equilibrium states of the encompassing shape, a software component 650 for selecting the usable physical object as a command object (user input interface object) to be used, where the command object (user interface object) can be a physical object (e.g., user selectable menu item, indicia, etc.) seen by the user through the XR device and when interacted with by the user indicates a command (input to a user input interface) to be performed by the XR device, and a software component for 660 assigning each of the one or more operating commands (inputs to user input interfaces) of the OS and / or application to each stable mechanical equilibrium state of the encompassing shape for the command object. The software component arrangement 600 further comprises a software component 670 for detecting a user interaction with the command object, a software component 680 for determining a resulting stable mechanical equilibrium state of the command object after user interaction and a software component 690 for executing the command (input to user input interface) assigned to the resulting stable mechanical equilibrium state. The software component arrangement 600 may further comprise software component(s) 695 for further functionality as discussed herein.

[0088] Various aspects of the XR device described above assume that the operations are able to select a physical object that has a number of stable mechanical equilibrium states which is preferably equal to the number of user input interfaces (also called "control states" and "required states (RS)") of the operating system (OS) program and / or application program of the XR device that are to be made available for a user to provide input to through interaction with the physical object. Some of these embodiments address a situation where the number of stable mechanical equilibrium states determined for an available physical object resembles, e.g. is the same as the number of required number of user input interfaces, referred to as a "perfect-fit scenario."

[0089] It is, however, expected that a general real-world environment, e.g. office desk, will not be occupied with and provide the XR user with physical objects that have a number of stable mechanical equilibrium states that matches the number of user input interfaces which need to be assigned. For example, the XR device operations may be adapted to assign a defined number of user selectable menu items to a same number of stable mechanical equilibrium states of a single physical object. However, the operations may be unable to identify a single physical object which is determined to have that number of stable mechanical equilibrium states.

[0090] In an illustrative operational example, an XR application has four menu options which are to be displayed via display 104 to the user and are supposed to be mapped to four stable mechanicalequilibrium states (also called "stable equilibrium states" for brevity) of a physical object which is present in the video from the image sensor 103. The user's desk has only four objects which are observed in the video: a ball, a triangle-based pyramid (tetrahedron) object, a smooth AirPods case (or something with that formfactor), and a uniform cube (i.e., a symmetric cube with equal lengths for depth, height, and width). In this example, the operations cannot select the ball because it is determined to not have the required number of stable equilibrium states. The operations should also not select the AirPods case because it is determined to have only two stable equilibrium states (top and bottom surfaces of the case) and would therefore not be suitable since only two of the four XR menus can be assigned to stable equilibrium states. In contrast, the operations can select the pyramid (tetrahedron) object because it can assign each of the four XR menu options to a different one of the four stable equilibrium states of the object, i.e., a one-to-one mapping between menu options and stable equilibrium states. However, there are typically a plurality of physical objects on a desk or other surface and which have other numbers of possible stable states than the exact number of user input interfaces which are to mapped for user input.

[0091] Various embodiments of the present disclosure are directed to operationally enabling the XR device to assign a determined number of user input interfaces of an OS and / or application program to a greater number of stable equilibrium states of a single physical object (referred to as "overfitted scenario") or to a select a set of physical objects which individually do not have a sufficient number of stable equilibrium states (referred to as "underfitted scenario") but that collectively provide the same or greater number of stable equilibrium states as the number of user input interfaces.

[0092] In the overfitted scenario, the number of stable equilibrium states of an available physical object is larger than the number of user input interfaces, e.g., XR-application menu entries, and the problem to solve here is how the operations shall assign (also called "map") the fewer number of user input interfaces to more than enough available number of stable equilibrium states for the corresponding physical object. For example, the operations should solve how to assign A menu entries among an object's B stable equilibrium states (where B> A), i.e., where not all of the stable equilibrium states B of a single object need to be assigned one of the A user input interfaces.

[0093] In the underfitted scenario, the number of stable equilibrium states of an available individual physical object is smaller than the number of user input interfaces, e.g., XR-application menu entries, and the problem to solve here is how the operations shall select a set of physical objects that are to be used as a combination that provides a combined number of stable equilibrium states that isgreater than the required number of user input interfaces. For example, the operations should solve how to assign A menu entries among a set of objects having a combined number B stable equilibrium states, where no individual one of those objects has a number of stable equilibrium states that is equal to or greater than A. In the underfitted scenario, the mapping cannot be obtained using only one physical object but more than one physical object can be selected for use as a set which collectively provide the required number of menu entries.

[0094] In these operational embodiments, a physical object's stable equilibrium states can be assigned to user input interfaces of the OS and / or application, such that a user can interact with the physical object relative to those stable equilibrium states to provide commands and other input. For example, an ordinary cube has six stable equilibrium states, but another physical object with certain convex surfaces may not be stable in every one of it possible stable equilibrium states. For example, a smartphone embedded in a protective case is typically stable while resting on its face and back surfaces on a tabletop, and although may be delicately balanced to stand up when positioned on one of its edge surfaces, the edge surfaces may not provide a sufficiently stable equilibrium state position for the smartphone to rest and for the operations to reliably assign a user input interface that stand up position.

[0095] Some further embodiments are directed to operations that can assign (map) at least one physical object's stable equilibrium state to a user input interface, e.g., an XR application user selectable menu item, when the physical object's number of equilibrium states are overfitting (e.g., number of stable equilibrium states > number of user input interfaces) or underfitting (e.g., number of stable equilibrium states < number of user input interfaces) with respect to the number of number of user input interfaces (e.g., XR application user selectable menu items) that need to be assigned.

[0096] Operational embodiments are provided which determine a physical object's stable equilibrium states and assign (map) those states to selected ones of the user input interfaces of the XR OS and / or application, e.g., the user selectable menu items, other indicia, etc. which are viewable by the user through the XR device.

[0097] Operations that can be performed by the controller 101 of the XR device 100 are now described with regard to the flowchart of operations shown in Figure 7. These operations can be used when the number of stable equilibrium states is equal to the number of user input interfaces that are to be assigned. Moreover, in accordance with further embodiments disclosed herein, these operations are adapted to be effectively used when there is an overfitted scenario or an underfitted scenario.Although operations are illustrated in Figure 7 as being performed in a sequence, it is to be understood that the order of operations can be changed without restriction, that two or more operations that are shown in sequence may be performed in parallel, and that two or more of the operations may be performed repetitively more often than other illustrated operations.

[0098] Referring to Figure 7, the operations detect 700 at least one physical object imaged in the video from the image sensor to be analyzed for use by a user to provide input to user input interfaces controlling display of content through the display. The operations identify 710 a number of stable mechanical equilibrium states of the at least one physical object. The operations determine 720 whether the number of stable number of stable mechanical equilibrium states is different than a number of the user input interfaces that are to be assigned.

[0099] When the number of stable mechanical equilibrium states is determined 720 to be the same as the number of the user input interfaces that are to be assigned (perfect fit scenario), the XR device may perform the operations described above relating to operational steps 550-590 in Figure 5.

[0100] In contrast, when the number of stable mechanical equilibrium states is determined 720 to be the same as the number of the user input interfaces that are to be assigned, the operations determine 730 assignments of individual ones of the user input interfaces to at least one of the stable mechanical equilibrium states of the at least one physical object based on a characteristic of the user input interface and / or a characteristic of the at least one of the stable mechanical equilibrium states. The operation then tracks 740, in the video from the image sensor 103, user interaction with one of the at least one physical object relative to one of the at least one of the stable mechanical equilibrium states of the interacted with one of the least one physical object. The operations responsively provide 750 input to the user input interface assigned to the one of the at least one of the stable mechanical equilibrium states of the interacted with one of the at least one physical object, based on the tracked user interaction.

[0101] The operation of the controller 101 of the XR device 100 (also referred to as "operation of the XR device 100" or "operation" for brevity) to identify 710 the number of stable mechanical equilibrium states of the at least one physical object, may include, for each of the at least one physical object, to: determine an encompassing shape for the physical object, wherein the encompassing shape comprises a plurality of geometrical shapes; and determine the number of stable mechanical equilibrium states of the physical object based on how many of the plurality of geometrical shapes havefeatures which are determined to be capable of supporting the physical object in a stable mechanical equilibrium state while resting on a physical surface.

[0102] The operations may be further configured to determine the number of stable mechanical equilibrium states of the physical object based on which of the plurality of geometrical shapes have features matching features of stored geometrical shapes defined as capable of supporting a physical object in a stable mechanical equilibrium state while resting on a physical surface.

[0103] In this manner, the XR device 100 can operate to enable a user to interact with one or more physical objects to provide corresponding input to assigned user input interfaces of an OS program and / or application program of the XR device 100. The operations can handle all of the above-described scenarios: perfect fit scenario, underfitted scenario, and overfitted scenario, to more reliably control operation of the XR device 100 under more diverse real-world situations where the user has the ability to use a wider variety of physical objects to provide input (e.g., control) to programs of the XR device 100.

[0104] These operational embodiments are first described below in the context of the overfitted scenario and are then described in the context of the underfitted scenario.

[0105] As explained above, in the overfitted scenario the number of stable equilibrium states of an available physical object is larger than the number of user input interfaces, e.g., XR-application menu entries. Various embodiments are therefore directed to assigning the fewer number of user input interfaces (e.g., menu entries) to more than enough available number of user input interfaces for the corresponding physical object.

[0106] In one embodiment, the operations to detect 700 at least one physical object imaged in the video from the image sensor 103, can include to: determine a required number of the user input interfaces that are to be assigned; and select one physical object for interaction with by the user to provide input to the user input interfaces, based on determining the number of stable mechanical equilibrium states of the one physical object is at least as great as the required number of the user input interfaces.

[0107] The operations can be configured to assign one of the user input interfaces to more than one of the stable mechanical equilibrium states. The controller 101 of the XR device 100 can respond to determining 720 the number of stable mechanical equilibrium states is greater than the required number of the user input interfaces that are to be assigned, by operating to assign 730 one of the user input interfaces to more than one of the stable mechanical equilibrium states.

[0108] When assigning one of the user input interfaces to more than one of the stable mechanical equilibrium states, the operations may seek to assign the user input interface to a spatially adjacent set of stable mechanical equilibrium states. Therefore, in the context of Figure 7, the operations to assign 730 one of the user input interfaces to more than one of the stable mechanical equilibrium states can include to: identify a set of at least two stable mechanical equilibrium states of the one physical object that satisfy a rule for being spatially adjacent; and assign the one of the user input interfaces to the identified set of at least two stable mechanical equilibrium states.

[0109] Alternatively or additionally, when assigning one of the user input interfaces to more than one of the stable mechanical equilibrium states, the operations may seek to assign the user input interface to a set of stable mechanical equilibrium states identified as requiring an amount of rotation and / or movement distance to transition between by user interaction with the one physical object which satisfies a rule. Therefore, in the context of Figure 7, the operations to assign 730 one of the user input interfaces to more than one of the stable mechanical equilibrium states can include to: identify a set of at least two stable mechanical equilibrium states of the one physical object that satisfy a rule for an amount of rotation and / or movement distance required to transition between by user interaction with the one physical object; and assign the one of the user input interfaces to the identified set of at least two stable mechanical equilibrium states.

[0110] The rule for the amount of rotation and / or movement distance required to transition between the at least two stable mechanical equilibrium states in the set by user interaction with the one physical object, may configure the operations to perform a define one of maximizing or minimizing the amount of rotation and / or movement distance required to transition between the at least two stable mechanical equilibrium states in the set by user interaction with the one physical object.

[0111] Alternatively or additionally, when assigning the user input interfaces to the stable mechanical equilibrium states, the operations may seek to space apart at least two of the assignments so that there is physical spacing between nearest ones of the stable mechanical equilibrium states assigned to different user input interfaces, e.g., to provide guard band spacing. Therefore, in the context of Figure 7, the controller 101 of the XR device 100 can respond to determining 720 the number of stable mechanical equilibrium states is greater than the required number of the user input interfaces that are to be assigned, by performing the determination 730 of the assignment of the user input interfaces to the stable mechanical equilibrium states while reserving at least one of the stable mechanical equilibrium states that is not assigned to any of the user input interfaces. The reserved atleast one of the stable mechanical equilibrium states is selected by the operations to be positionally located on the one physical object between a spaced apart pair of two other stable mechanical equilibrium states that are mapped to two different user input interfaces.

[0112] Some further embodiments are directed to using relative probabilities that respective ones of the user input interfaces will be selected next for input by the user.

[0113] In the context of Figure 7, the controller 101 of the XR device 100 can respond to determining 720 the number of stable mechanical equilibrium states is greater than the required number of the user input interfaces that are to be assigned, by determining 730 assignments of the user input interfaces to the stable mechanical equilibrium states of the one physical object based on comparison of probabilities that respective ones of the user input interfaces will be selected next by the user to receive user input while the XR device is operating in a present operational state. The operations may, for each of the user input interfaces, compute the probability based on values indicating likelihood of the XR device transitioning from the present operational state to another allowed operational state by user input provided to the respective user input interface.

[0114] The operation of the controller 101 of the XR device 100 to determine assignments 730 of the user input interfaces to the stable mechanical equilibrium states of the one physical object, may include to assign one of the user input interfaces having the greatest probability to a greater number of the stable mechanical equilibrium states than another one of the user input interfaces have a lowest probability.

[0115] Thus, for example, when a set of user-selectable items are to be displayed on the display 104 of the XR device 100, the operations may be configured to assign 730 more, e.g., number X, stable mechanical equilibrium states of the one physical object (e.g., two states corresponding to physically adjacent surfaces, such as adjacent sides of a cube) to one of the user-selectable items that is determined to be most likely to be selected by the user based on prior user-selections of other items by the user and / or based on probabilities that have been defined for which operations states the user will want to transition to next from a present operational state of the OS and / or application program of the XR device 100. In contrast, the operations may be configured to assign 730 less, e.g., number Y where Y< X, stable mechanical equilibrium state(s) of the one physical object to another one of the user-selectable items that is determined to be less likely to be selected by the user based on prior userselections of other items by the user and / or based on probabilities that have been defined for whichoperations states the user will want to transition to next from a present operational state of the OS and / or application program of the XR device 100.

[0116] The transition probabilities may be inferred based on characteristics of the user input interfaces for the operating system program and / or application program, and may for example be based on Markov chain transition probabilities. The operations may, for example, determine:• %ratio Al of highest transition probability between a first and a second operational state of the program,• %ratio A2 of second highest transition probability between a first and a third operational state, • %ratio A3 of third highest transition probability between a first and a fourth operational state, etc.• %ratio Ai of i:th transition probability... etc.• % ratio A_remainder that collects other less used transition probabilities (where A_remainder > resolve_individual_states threshold. The operations can collect rarely used states into a separate group state and unfold that collection group state into a new loop, where respective Al, A2, A3... Ai [0..100%].

[0117] The operations can calculate a number of AS-allocations (nASa) that a highest transition probability Al corresponds to among control object AS, such as:• nASa_l = ceil(Al* AS) (... or alternatively nASa1 = floor(A1* AS),or "round")typically a floor-truncation will act conservatively and conserve %fraction-space allocations to allow more smaller fraction to perhaps get state allocations.• nASa_2 = ceil(A2* AS), nASa_3 = ceil(A3* AS),...• nAS _remainder= ceil(A_remainder* AS).

[0118] The operations can render (in the aspect of executing state-representing allocations to the control object), according to determined respective nASaJ, such as:• a number of states "nASa_l" are allocated to the control object representing the most probable transition probability from a starting state S.• a number of states "nASa_2" are allocated to the control object representing the most probable transition probability from a starting state S.• Etc.

[0119] The inclusion of " A_remainder" is intended to gather "other" less used transition probabilities associated with digital application interaction that are rather infrequently used, but nevertheless still possible.

[0120] Operations may compare to a hierarchical set of user input interfaces, e.g., menu items with some having a logical hierarchical association (e.g., menu-unfold) that are present in today's menu solutions to save space and improve most common user applications interactions so to unhide less / rarely / never used user input interfaces of application functions, etc., as the user interacts with certain application icon / objects and / or hovers mouse pointer over longer than some predefined / user-specific threshold.

[0121] The A_remainder can be determined so as to gather less frequently user states, and be assigned a certain control object accessible state. As the user interacts with the physical object's assigned A_remainder state, the operations responsively select, according to a defined rule (e.g., a top-down, alphabetically, or otherwise predefined hierarchical association), the remainder of possible required stable mechanical states that are not yet subject to user's active transitions can be assigned to remaining user input interfaces.

[0122] Alternatively or additionally, when the operations determine 720 that the number of stable mechanical equilibrium states is greater than the required number of the user input interfaces that are to be assigned, the operations may determine 730 assignments of the user input interfaces to the stable mechanical equilibrium states of the one physical object based on comparison of relative sizes of physical object surfaces, which correspond to the determined mechanical equilibrium states of the physical object, to the sizes of user-selectable items that will be displayed to the user to represent the corresponding user input interfaces. Thus, for example, when a set of user-selectable items are to be displayed on the display 104 of the XR device 100, the operations may determine assignments of the user input interfaces to the stable mechanical equilibrium states of the one physical object based on comparison of sizes of physical object surfaces, which correspond to the determined mechanical equilibrium states, to the sizes of user-selectable items that will be displayed to the user to represent the corresponding user input interfaces.

[0123] Thus, for example, larger ones of the displayed menu items (or other indicia) can be mapped to larger ones of the physical object surfaces of the object, and smaller ones of the displayed menu items (or other indicia) can be mapped to smaller ones of the physical object surfaces of the object. In one embodiment, the largest displayed menu item (or other indicia) is mapped to the largest physicalobject surface of the object (which corresponds to one of the mechanical equilibrium states) and the smallest displayed menu item (or other indicia) is mapped to the smallest physical object surface of the object (which corresponds to another one of the mechanical equilibrium states).

[0124] Operational embodiments are now described in the context of the underfitting scenario.

[0125] As explained above, in the underfitting scenario the number of stable equilibrium states of an available physical object is smaller than the number of user input interfaces, e.g., XR-application menu entries. Various embodiments are therefore directed to assigning the larger number of user input interfaces (e.g., menu entries) to a plurality of objects which each individual may have fewer number of stable equilibrium states than the number of user input interfaces, but collectively provide the same or more stable equilibrium states than the number of user input interfaces.

[0126] To manage the underfitting scenario, the operation to detect 700 at least one physical object imaged in the video from the image sensor to be analyzed for use by the user to provide input to user input interfaces controlling display of content through the display, comprises to determine a required number of the user input interfaces that are to be assigned, and to select at least two physical objects for interaction by the user to provide input to the user input interfaces, based on determining that a combination of the number of stable mechanical equilibrium states of the at least two physical objects is at least as great as the required number of the user input interfaces.

[0127] In a further operational embodiment, the operations are configured to select a set of more than one physical object for assignment of their collective stable mechanical equilibrium states, only when no single physical object in the video from the image sensor 103 is determined to have a number of stable mechanical equilibrium states that is at least as great as the required number of the user input interfaces of the application. In the context of Figure 7, the operation to select 700 at least two physical objects for interaction by the user to provide input to the user input interfaces (based on determining that the combination of the number of stable mechanical equilibrium states of the at least two physical objects is at least as great as the required number of the user input interfaces), includes to determine that no physical object in the video has a number of stable mechanical equilibrium states that is at least as great as the required number of the user input interfaces. Thus, when no one physical object has a sufficient number of stable mechanical equilibrium states, the operation can select more than one physical object (pursuant to one or more selection rules) for assignment 730 of their collective stable mechanical equilibrium states to the required user input interfaces in order to operationally allow the user to interact with 740 those physical objects to provide input to the user input interfaces.

[0128] In a further operational embodiment, the operations are configured to use more than one physical object based on receiving a user selection of a plurality of physical objects in the video for assignment to the user input interfaces of the application. In the context of Figure 7, the operation to select 700 at least two physical objects for interaction by the user to provide input to the user input interfaces (based on determining that the combination of the number of stable mechanical equilibrium states of the at least two physical objects is at least as great as the required number of the user input interfaces), includes to determine receiving an indication of the user's selection of the at least two physical objects to be used for assignment to the user input interfaces. Thus, for example, the user may select (e.g., by point toward or physically touching within view of the imaging sensor 103) more than two physical objects viewed in the display 105 of the XR device 100. The operations can respond to those selections by selecting 700 those user-selected physical objects for assignment 730 of user input interfaces to operationally allow the user to interact with 740 those physical objects to provide input to the user input interfaces.

[0129] Embodiments are described below in the context of determining user preferred interaction.

[0130] The operations may track a user's interaction with a physical object when providing input to assigned user input interfaces, and determine therefrom over-time a preferred way the user wants to interact with the stable mechanical equilibrium states of that particular physical object or more generally with any similarly shaped physical object.

[0131] For example, a user being left-handed may prefer to interact with a physical object along a specific "left-to-right"-type of path as that holds certain biomechanical advantages.

[0132] The operations may determine that the user in respect to e.g., a physical object having two stable mechanical equilibriums states always, often, or rarely interacts from a first-to-second state in a clockwise manner, but in contrast respectively rarely, often, or always executes the opposite pattern direction.

[0133] Then the operations can deduct a preferred toggling / flip direction to be re-used for other more complex control physical object accessible state assignments to user input interfaces for input by the user.

[0134] Teachings that provide for detecting physical objects that are not considered usable physical objects (e.g., objects that include a number of states that is less than a number of operation commands compatible as inputs to a current operational state of an application) are discussed below. However,some embodiments discussed above may be used as additional / alternative embodiments to the embodiments discussed below.

[0135] For a scenario with an XR application menu to be interacted with, there are different possible approaches: a user may make use of a physical controller device either connected to the main platform entity via cable, Bluetooth, Wi-Fi or even with recent developments some local cellular D2D interface (or some other wireless interface), or a user may use a gesture-based Ul where user's hands and associated gestures act as " Ul controllers" without actually having any physical object grabbed. However, another approach is available where an XR system scans, evaluates and selects a physical object as anchor object to render an in-application relevant texture onto; in this way the user may benefit from the biological evolution of having eye-muscle memory aligned interacting with something grabbable in combination with manipulating and texturizing digital pixels in an XR environment. Then a physical object with relevant physical properties (e.g., shape, mass, center of mass, inertia, etc.) could be enriched with in-application digital attributes.

[0136] In an example, the XR device may project an in-application texture for an in-application controller (e.g., a game pad) onto a physical object that a user would use to play or otherwise interact with in an application. For example, a physical object (e.g., a large-size spring onion with sufficiently correct formfactor, moment of inertia, etc.) may be selectable (by a user of by the XR device) as physical-grab-object in a sports tennis game. However, to make the overall gaming experience better and perhaps more realistic, it may be beneficial not to show the spring onion itself towards the gaming user, but instead in an overlay aspect, rendering the in-application tennis-racket texture onto the user's in-game view towards the spring onion.

[0137] It may be beneficial to the user's experience that the selected physical texture anchor object holds some physical properties (e.g. inertia, mass, center of mass, center of inertia) as the in-application controller (e.g., tennis racket), otherwise the XR device may not generate a correct alignment between the physical object and the in-application controller and / or generate accurate feedback to the user.

[0138] This is especially true when the physical object includes a number of states that is less than a number of operation commands compatible as inputs to a current operational state of the application. In these instances, it may be beneficial for the XR device to map or re-map operation commands of an in-application controller to states of a physical object.

[0139] Figure 8A shows an example of an in-application texture 804a for an in-application controller 800 that includes eight operation commands, according to some embodiments of the presentdisclosure. Figure 8B shows another example of an in-application texture 804b for an in-application controller 802 that includes three operation commands, according to some embodiments of the present disclosure.

[0140] Figure 9 shows a flowchart of operations by an XR device (e.g., XR device 100) for mapping states of a physical object interacted with by a user to provide operation command inputs to an XR application, according to some embodiments of the present disclosure. Operations of Figure 9 are discussed below with references to the examples of Figures 8A-8B.

[0141] In operation 900, the XR device detects a physical object in a field of view of an image sensor (e.g., image sensor 103). In the example of Figures 8A-8B, the XR device may scan the field of view including a table with one or more objects on it and detect an physical object that includes a three-dimensional triangle block.

[0142] The XR device (in operation 902) determines a number of states of the physical object that satisfies a rule for a user to interact with the physical object to provide operational command inputs to an application. For example, the rule may define that only stable mechanical equilibrium states satisfy the rule and the XR device may determine that the three-dimensional triangle block has five surfaces, three of which are stable mechanical equilibrium states and two of which are not stable mechanical equilibrium states. Thus, the XR device may determine that the number of states of the physical object that satisfy the rule is three.

[0143] In some embodiments, the determined number of states of the physical object comprises a number of stable mechanical equilibrium states. In some additional, or alternative, embodiments, the determined number of states of the physical object comprises another type of state that satisfies a rule for a user to interact with an object to provide operational command inputs to an application. These another type of states are described above and may further include at least one of a type of material of the physical object, surface macro / microstructure, texture, weight, size, temperature, etc.

[0144] In operation 904, the XR device determines whether the number of determined states is less than a number of operation commands compatible as inputs to a current operational state of the application. If the XR device determines that the number of determined states is not less than a number of operation commands, then the XR device goes to operation 702 of Figure 7. Alternatively, responsive to this determination, the XR device may terminate the application and generate an error message to indicate that the field of view does not include a usable object or the XR device may pause theapplication and generate a message to indicate that more objects are needed in the field of view or that the user should change the field of view.

[0145] If the XR device determines that the number of determined states is less than a number of operation commands, then the XR device goes to operation 906.

[0146] In the example of Figures 8A-8B, a current operational state of an application includes a user entering a vehicle to drive the vehicle in an XR environment. The vehicle has gears that the user can change in the XR environment by a user adjusting an in-application controller 800 displayed to the user as in-application texture 804a. The application may have eight operation commands compatible as inputs to the current operational state of the application. The eight operation commands, in this example, may include six forward gears (gears 1 through 6), a neutral gear, and a reverse gear (gear R) for the in-application vehicle. The XR device may determine that the number of states of the physical object (3 states) is less than the number of operation commands (eight operation commands) that are compatible as inputs to the current operational state of the application.

[0147] In operation 906, responsive to determining (in operation 904) that the number of determined stable mechanical equilibrium states is less than the number of operation commands of the application, the XR device selects a subset of the operation commands of the application. In the example of Figure 8A-8B, responsive to determining that there are three states which is less than eight operation commands of the application, the XR device may select a subset of the operation commands of the application. The XR device may select three most important operation commands (e.g., the three most prioritized operational commands) in a hierarchy table of a plurality of operation commands. The XR device may alternatively, or in combination with some of the most prioritized operation commands, select operation command(s) that are mandatory for inputs to a current operation state of the application.

[0148] In the example of Figure 8A-8B, this may result in the XR device selecting one forward gear, one neutral gear, and one reverse gear as the subset of operation commands.

[0149] In operation 908, the XR device assigns each operation command of the subset to a different state of the physical object. In the example, the XR device may assign the forward gear to a first stable mechanical equilibrium state of the three-dimensional triangle, the neutral gear to a second stable mechanical equilibrium state of the three-dimensional triangle, and the reverse gear to a third stable mechanical equilibrium state of the three-dimensional triangle. This can allow the user to interact withthe three-dimensional triangle to input the execution of the different operation commands (e.g., changing of gear that the in-application vehicle uses in the XR environment).

[0150] The XR device may determine whether the physical object is in a state that corresponds to an assigned operation command. Additionally, the XR device may, responsive to determining that the physical object is in the state that corresponds to the assigned operation command, execute the assigned operation command. For example, if the user interacts with the three-dimensional triangle into the first stable mechanical equilibrium state of the three-dimensional triangle, the XR device may determine that the physical object is in the first stable mechanical equilibrium state and the XR device executes a selection of the forward gear in the application. Similarly, if the user then interacts with the three-dimensional triangle into the third stable mechanical equilibrium state of the three-dimensional triangle, the XR device may determine that the physical object is in the third stable mechanical equilibrium state and the XR device executes a selection of the reverse gear in the application.

[0151] In some embodiments, the XR device generates an in-application texture for an inapplication controller that is associated with a number of in-application controllable operations that correspond to the number of operation commands of the subset. In some additional or alternative embodiments, the XR device controls a display (e.g., display 104) to render the in-application texture on the display with each operation command of the subset mapped to the assigned different one of the states of the physical object. In the example of Figure 8A-8B, the XR device generates the in-application texture 804b for the in-application controller 802 that is associated with a number of in-application controllable operations (forward gear, neutral gear, and reverse gear) that correspond to the number of operation commands of the subset. The XR device controls a display (e.g., display 104) to render the inapplication texture 804b on the display with each operation command of the subset mapped to the assigned different one of the states of the physical object.

[0152] The in-application texture 804b may include the forward gear displayed as a block D, the neutral gear displayed as a block N, and the reverse gear displayed as a block R, with a shifter (indicated by the circle) indicating which operation command (in this example which gear) is currently selected (i.e., which of the three states the physical object is currently in).

[0153] The in-application texture may include an object shape, object color, and / or object surface appearance.

[0154] The XR device may further control a feedback generator to provide the user an indication of a number of in-application controllable operations that correspond to the number of operationcommands of the subset. The feedback generator may include at least one of a tactile feedback generator, haptic feedback generator, thermal feedback generator, and audio feedback generator. For example, the tactile feedback generator can be used to generate a tactile sensation (e.g., a vibration) that is felt by the user through the physical object or another component (e.g., a user worn XR suit, user worn XR gloves, etc.) that provides feedback to the user regarding the in-application texture (e.g., shape of the in-application texture, surface texture of the in-application texture, etc.). In another example, the thermal feedback generator could render a thermal signature / pattern associated with the in-application texture. If the user is wearing XR gloves, the XR device may control the thermal feedback generator to generate a warm or cold sensation to the user based on the in-application texture in the application.

[0155] In some embodiments, responsive to determining that the number of determined states is less than the number of operation commands of the application, the XR device generates an indication for display, through a display (e.g., display 104), indicating at least one of: a type of physical object that includes a number of states that includes the number of operation commands of the application; and a number of states a physical object is to include, to have the number operation commands of the application. In the example of Figure 8A-8B, the XR device could generate an indication for display indicating at least one of a type of object (e.g., an octagon) that includes eight states and the number eight since the number of operation commands compatible as inputs to the current operational state of the application includes eight operation commands.

[0156] In some additional or alternative embodiments, responsive to determining that the number of determined states is less than the number of operation commands of the application, the XR device generates an indication for display, through the display (e.g., display 104), indicating that operation commands in the application are limited. In some of these embodiments, this includes generating text for display that indicates that the operation commands in the application are limited. In other ones of these embodiments, the generation of the indication for display, through the display, indicating that operation commands in the application are limited includes the XR device selecting an in-application color and / or in-application texture for an in-application controller which is unique to the determination that the number of determined states is less than the number of operation commands of the application. The in-application color and / or in-application texture may be selected among a defined group of in-application colors and / or in-application textures based on an amount of difference between the number of determined states and the number of operation commands of the application. For example, if the amount of difference between the number of determined states and the number ofoperation commands of the application is one an in-application color of yellow may be selected by the XR device. In another example, if the amount of difference between the number of determined states and the number of operation commands of the application is two an in-application color of orange may be selected by the XR device. In yet another example, if the amount of difference between the number of determined states and the number of operation commands of the application is three or more an inapplication color of red may be selected by the XR device.

[0157] Alternatively, the in-application color and / or in-application texture is based on the number the operation commands are reduced to. For example, an in-application color of yellow may be selected for operation commands that are reduced to three operation commands regardless of the total number of operation commands that are compatible as inputs to a current operational state of the application.

[0158] Alternatively or additionally, responsive to determining that the number of determined states is less than the number of operation commands of the application, the XR device may generate at least one of a tactile indication, haptic indication, thermal indication, and audio indication that indicates that operation commands in the application are limited. For example, the XR device may generate a warm thermal pattern through a user worn XR glove when the user interacts with an in-application controller that includes a number of operation commands that are limited compared to a total number of operation commands compatible as inputs to a current operational state of the application or are limited in that they are a subset of the total operation commands compatible as inputs to a current operational state of the application. In another example, the XR device may generate an audio indication that auditorily indicates that operation commands in the application are limited as inputs to a current operational state of the application. The generated indication may vary based on an amount of difference between the number of determined states and the number of operation commands of the application or based on the number of operation commands the number of operation commands is reduced to. For example, a higher tactile sensation or louder audio is generated the larger the difference between the number of determined states and the number of operation commands of the application, or vice versa.

[0159] In some embodiments, the XR device generates a message for transmission to a server that includes an indication of at least one of: the subset of operation commands that are assigned to the physical object; and an indication that less than all of the operation commands are assigned to the physical object. For example, if the XR device is a XR HMD, the XR HMD may generate a message for transmission to a server that includes an indication that a specific subset of operation commands havebeen assigned to a specific object. In another example, a server entity could be performed by another XR device (e.g., in platform setup in a two-person gaming scenario) where one of the XR devices acts as a game-hosting server and the XR device connects to it. The XR device may generate a message for transmission to a server or to the another XR device to indicate that a specific subset of operation commands is assigned to an physical object the XR device is using and / or to indicate that less than all of the operation commands are assigned to the physical object.

[0160] This may provide the benefit of informing the server or the another XR device that the XR device is limited or enhanced by assigning a subset or less than all of the operation commands to an physical object. In the example of Figure 8A-8B, a user of the XR device may benefit from having three operation commands rather than eight because the user may need to shift gears less than another XR device user. It may be beneficial to provide this information to the server for setting up (e.g., grouping users) gaming lobbies or for informing the other XR devices in a lobby that one or more of the users in the lobby have an advantage / disadvantage. A fully capable user (user using all the available operation commands compatible as inputs to a current operational state of the application) may face more ingame complexity while being more realistic, compared to a user using a reduced-control functionality (subset of all the available operation commands compatible as inputs to a current operational state of the application) that may be simpler to use but being less realistic. Nevertheless, it may be beneficial for a server to be able to group / cluster user-teams with respect to their respective full / reduced operation command capabilities.

[0161] In instances where the server sets up a lobby or groups XR devices based on a subset of operation commands that is assigned to an physical object, the lobby may group XR devices that include the same subset of operation commands or similar subset of operation commands. The similar subset of operation commands may include subsets where the difference between the subsets includes a threshold number of operation commands (e.g., two or less) or the type of subset of operation commands is similar (e.g., the subsets both converted an in-application controller from a manual gearbox to an automatic gearbox).

[0162] In some embodiments where the XR device comprises a server networked to a XR head mounted device (HMD) or networked to a handheld XR device, the XR device groups into a cooperative multi-user XR environment the XR HMD or the handheld XR device running the application with one or more other XR HMDs or other handheld XR devices that are running the application and have operation commands assigned to states of a physical object. The grouping of the XR HMD or the handheld XRdevice with the one or more other XR HMDs or other handheld XR devices may be based on the one or more other XR HMDs or other handheld XR devices having a same subset of operation commands assigned to states of the physical object. The grouping may additionally be based on a user's skill level, achievements, etc. in the application.

[0163] In some of these embodiments, where the XR device comprises a server networked to a XR head mounted device (HMD) or networked to a handheld XR device, the XR device transmits to the XR HMD or the handheld XR device an indication of the operation commands the one or more other XR HMDs or other handheld XR devices are running the application with. This may provide the benefit of alerting the user of the XR HMD or the handheld XR device that certain operation commands are being used (or not used) by other users. Thus, the user of the XR HMD or the handheld XR device is alerted to potential advantages or disadvantages of other users.

[0164] The XR device may further detect another physical object in the field of view, determine a number of states of the another physical object. The XR device may generate an indication provided for display through a display (e.g., display 104) that indicates that the another physical object is not suitable for use to control operation commands of the application based on comparison of the number of operation commands of the application to the determined number of states of the another physical object.

[0165] In some additional embodiments, the XR device detects a collection of physical objects in the field of view, where the collection includes the physical object and at least one other physical object, and determines a parameter of each physical object in the collection. The parameter includes at least one of a size, shape, surface texture, weight, and temperature of the respective physical object. The XR device may select the OBJ from among the collection of physical objects based on the determined parameter of the physical object satisfying a rule for usability for tracking, through an image sensor (103), user input for operation commands of the application. This may provide the benefit of selecting an physical object that is more suitable for use for a certain application, certain in-application controller, and / or is more suitable for a user to grip (e.g., fits within a user's hand or is a weight that would provide a sensation to the user that would be similar to the user holding the in-application controller in the real world). For example, the XR device may select a physical object that has course exterior surfaces and medium size that is easier for the user to grip over a physical object that has smooth exterior surfaces that are slick and a large size that is not easy for the user to grip. This determination may take into account a determined size of the user's hand when detected in the field of view.

[0166] In some of these embodiments, the XR device detects a collection of physical objects in the field of view, where the collection includes the physical object and at least one other physical object, and obtains an object history of the collection of physical objects. The object history may include data indicating at least one of: prior use of each physical object in the collection to provide control input for operation commands in the application, usage preferences compared to other physical objects, an amount of time the physical object has been used, time of day the physical object was used, etc. The XR device may select the physical object from among the collection of physical objects based on the obtained object history.

[0167] For example, a first object with a slippery surface texture and smaller size may have been previously selected but is evaluated (by the XR device or another device) to have an in-application control-action precision / accuracy worse than a threshold value, and based on that, that the first "slippery and small" object is given a lower priority of use classification. A second object with a slightly larger size and with a more grippy physical texture may have previously rendered better control-action precision / accuracy, and is by the system therefore associated with a higher priority classification. The XR device may select among a set of preferred-classified objects to select a historically evaluated "best" or "preferred" object for a certain application or current operational state of the application. If the first object includes a same number of states as a second object, the second object may thus be selected based on the objects' history.

[0168] In a scenario where several users are making use of same physical environment, it follows that different physical XR users may have their respective "preferred" control object or set of objects with respect to specific XR applications. The XR device may factor in other user's object history for users that are in the same real-world environment as the user of the XR device.

[0169] The XR device may further assign a different haptic pattern to be generated by a haptic generator or audio to be played through a speaker, to different states of the physical object that are assigned to the operation commands of the subset. The haptic pattern or audio may be assigned based on the state of the physical object or based on the operation command that is assigned to the state of the physical object. For example, a first audio may be assigned to a first state of an physical object based on the first state being a specific shape, orientation, pattern, etc. Additionally, or alternatively, a first haptic pattern is assigned to a second state of the physical object based on the second state being assigned (by the XR device) a first operation command. Each determined state may be assigned a hapticpattern and / or audio, or a subset of the determined states may be assigned a haptic pattern and / or audio.

[0170] While applications such as gaming application are discussed throughout, it should be understood that embodiments of the present disclosure can also be applied to in-application menus, other in-application controllers, and other types of applications such as social media applications, web browser applications, etc.

[0171] Further definitions and embodiments are described below.

[0172] In the above-description of various embodiments of present inventive concepts, it is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of present inventive concepts. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which present inventive concepts belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense expressly so defined herein.

[0173] When an element is referred to as being "connected", "coupled", "responsive", or variants thereof to another element, it can be directly connected, coupled, or responsive to the other element or intervening elements may be present. In contrast, when an element is referred to as being "directly connected", "directly coupled", "directly responsive", or variants thereof to another element, there are no intervening elements present. Like numbers refer to like elements throughout. Furthermore, "coupled", "connected", "responsive", or variants thereof as used herein may include wirelessly coupled, connected, or responsive. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Well-known functions or constructions may not be described in detail for brevity and / or clarity. The term "and / or" includes any and all combinations of one or more of the associated listed items.

[0174] It will be understood that although the terms first, second, third, etc. may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another element / operation. Thus, a first element / operation in some embodiments could be termed a second element / operation in other embodiments without departing from the teachings of present inventive concepts. The same referencenumerals or the same reference designators denote the same or similar elements throughout the specification.

[0175] As used herein, the terms "comprise", "comprising", "comprises", "include", "including", "includes", "have", "has", "having", or variants thereof are open-ended, and include one or more stated features, integers, elements, steps, components or functions but does not preclude the presence or addition of one or more other features, integers, elements, steps, components, functions or groups thereof. Furthermore, as used herein, the common abbreviation "e.g.", which derives from the Latin phrase "exempli gratia," may be used to introduce or specify a general example or examples of a previously mentioned item, and is not intended to be limiting of such item. The common abbreviation "i.e.", which derives from the Latin phrase "id est," may be used to specify a particular item from a more general recitation.

[0176] Example embodiments are described herein with reference to block diagrams and / or flowchart illustrations of computer-implemented methods, apparatus (systems and / or devices) and / or computer program products. It is understood that a block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by computer program instructions that are performed by one or more computer circuits. These computer program instructions may be provided to a processor circuit of a general purpose computer circuit, special purpose computer circuit, and / or other programmable data processing circuit to produce a machine, such that the instructions, which execute via the processor of the computer and / or other programmable data processing apparatus, transform and control transistors, values stored in memory locations, and other hardware components within such circuitry to implement the functions / acts specified in the block diagrams and / or flowchart block or blocks, and thereby create means (functionality) and / or structure for implementing the functions / acts specified in the block diagrams and / or flowchart block(s).

[0177] These computer program instructions may also be stored in a tangible computer-readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instructions which implement the functions / acts specified in the block diagrams and / or flowchart block or blocks. Accordingly, embodiments of present inventive concepts may be embodied in hardware and / or in software (including firmware, resident software, micro-code,etc.) that runs on a processor such as a digital signal processor, which may collectively be referred to as "circuitry," "a module" or variants thereof.

[0178] It should also be noted that in some alternate implementations, the functions / acts noted in the blocks may occur out of the order noted in the flowcharts. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality / acts involved. Moreover, the functionality of a given block of the flowcharts and / or block diagrams may be separated into multiple blocks and / or the functionality of two or more blocks of the flowcharts and / or block diagrams may be at least partially integrated. Finally, other blocks may be added / inserted between the blocks that are illustrated, and / or blocks / operations may be omitted without departing from the scope of inventive concepts. Moreover, although some of the diagrams include arrows on communication paths to show a primary direction of communication, it is to be understood that communication may occur in the opposite direction to the depicted arrows.

[0179] Many variations and modifications can be made to the embodiments without substantially departing from the principles of the present inventive concepts. All such variations and modifications are intended to be included herein within the scope of present inventive concepts. Accordingly, the above disclosed subject matter is to be considered illustrative, and not restrictive, and the appended examples of embodiments are intended to cover all such modifications, enhancements, and other embodiments, which fall within the spirit and scope of present inventive concepts. Thus, to the maximum extent allowed by law, the scope of present inventive concepts is to be determined by the broadest permissible interpretation of the present disclosure including the following examples of embodiments and their equivalents, and shall not be restricted or limited by the foregoing detailed description.

Claims

CLAIMS:

1. An Extended Reality, XR, device (100) comprising a controller (101), wherein the controller (101) is operative to:detect a physical object, OBJ, in a field of view, FOV, of an image sensor (103);determine a number of states of the OBJ that satisfies a rule for a user to interact with the OBJ to provide operational command inputs to an application;determine whether the number of determined states is less than a number of operation commands compatible as inputs to a current operational state of the application;responsive to determining that the number of determined states is less than the number of operation commands of the application, select a subset of the operation commands of the application; andassign each operation command of the subset to a different state of the OBJ.

2. The XR device (100) according to claim 1, wherein the controller (101) is further operative to: determine whether the OBJ is in a state that corresponds to an assigned operation command; andresponsive to determining that the OBJ is in the state that corresponds to the assigned operation command, execute the assigned operation command.

3. The XR device (100) according to any of claims 1 to 2, wherein the controller (101) is further operative to:generate an in-application texture for an in-application controller that is associated with a number of in-application controllable operations that correspond to the number of operation commands of the subset; andcontrol a display (104) to render the in-application texture on the display (104) with each operation command of the subset mapped to the assigned different one of the states of the OBJ.

4. The XR device (100) according to claim 3, wherein the in-application texture comprises an object shape, object color, and / or object surface appearance.

5. The XR device (100) according to any of claims 1 to 4, wherein the controller (101) is further operative to:control a feedback generator to provide the user an indication of a number of in-application controllable operations that correspond to the number of operation commands of the subset, wherein the feedback generator comprises at least one of a tactile feedback generator, haptic feedback generator, thermal feedback generator, and audio feedback generator.

6. The XR device (100) according to any of claims 1 to 2, wherein the controller (101) is further operative to:responsive to determining that the number of determined states is less than the number of operation commands of the application, generate an indication for display, through a display (104), indicating at least one of:a type of physical object that includes a number of states that includes the number of operation commands of the application; anda number of states a physical object is to include, to have the number operation commands of the application.

7. The XR device (100) according to any of claims 1 to 2, further comprising a display (104), wherein the controller (101) is further operative to:responsive to determining that the number of determined states is less than the number of operation commands of the application, generate an indication for display, through the display (104), indicating that operation commands in the application are limited.

8. The XR device (100) according to claim 7, wherein the generation of the indication for display, through the display (104), indicating that operation commands in the application are limited, comprises selecting an in-application color and / or in-application texture for an in-application controller which is unique to the determination that the number of determined states is less than the number of operation commands of the application.

9. The XR device (100) according to claim 8, wherein the in-application color and / or inapplication texture is selected among a defined group of in-application colors and / or in-applicationtextures based on an amount of difference between the number of determined states and the number of operation commands of the application.

10. The XR device (100) according to any of claims 1 to 1, wherein the controller (101) is further operative to:responsive to determining that the number of determined states is less than the number of operation commands of the application, generate at least one of a tactile indication, haptic indication, thermal indication, and audio indication that indicates that operation commands in the application are limited.

11. The XR device (100) according to any of claims 1 to 10, wherein the controller (101) is further operative to:generate a message for transmission to a server that includes an indication of at least one of: the subset of operation commands that are assigned to the OBJ; andan indication that less than all of the operation commands are assigned to the OBJ.

12. The XR device (100) according to any of claims 1 to 11, wherein the controller (101) is further operative to:detect another physical object in the FOV;determine a number of states of the another physical object; andgenerate an indication provided for display through a display (104) that indicates that the another physical object is not suitable for use to control operation commands of the application based on comparison of the number of operation commands of the application to the determined number of states of the another physical object.

13. The XR device (100) according to any of claims 1 to 11, wherein the controller (101) is further operative to:detect a collection of physical objects in the FOV, wherein the collection includes the OBJ and at least one other physical object;determine a parameter of each physical object in the collection, wherein the parameter includes at least one of a size, shape, surface texture, weight, and temperature of the respective physical object; andselect the OBJ from among the collection of physical objects based on the determined parameter of the OBJ satisfying a rule for usability for tracking, through an image sensor (103), user input for operation commands of the application.

14. The XR device (100) according to any of claims 1 to 11, wherein the controller (101) is further operative to:detect a collection of physical objects in the FOV, wherein the collection includes the OBJ and at least one other physical object;obtain an object history of the collection of physical objects, wherein the object history includes data indicating prior use of each physical object in the collection to provide control input for operation commands in the application; andselect the OBJ from among the collection of physical objects based on the obtained object history.

15. The XR device (100) according to any of claims 1 to 14, wherein the controller (101) is further operative to:assign a different haptic pattern to be generated by a haptic generator or audio to be played through a speaker, to different states of the OBJ that are assigned to the operation commands of the subset.

16. The XR device (100) according to any of claims 1 to 15, wherein the XR device (100) comprises a XR head mounted device (HMD) or a handheld XR device.

17. The XR device (100) according to any of claims 1 to 16, wherein the XR device (100) comprises a server networked to a XR head mounted device (HMD) or networked to a handheld XR device.

18. The XR device (100) according to claim 17, further comprising:grouping into a cooperative multi-user XR environment the XR HMD or the handheld XR device running the application with one or more other XR HMDs or other handheld XR devices that are running the application and have operation commands assigned to states of a physical object.

19. The XR device (100) according to claim 18, wherein grouping the XR HMD or the handheld XR device with the one or more other XR HMDs or other handheld XR devices is based on the one or more other XR HMDs or other handheld XR devices having a same subset of operation commands assigned to states of the physical object.

20. The XR device (100) according to claim 18, further comprising:transmitting to the XR HMD or the handheld XR device an indication of the operation commands the one or more other XR HMDs or other handheld XR devices are running the application with.

21. The XR device (100) according to any of claims 1 to 20, wherein the determined number of states of the OBJ comprises a number of stable mechanical equilibrium states.

22. A method by an Extended Reality, XR, device comprising a controller, the method comprising:detecting (900) a physical object, OBJ, in a field of view, FOV, of an image sensor; determining (902) a number of states of the OBJ that satisfies a rule for a user to interact with the OBJ to provide operational command inputs to an application;determining (904) whether the number of determined states is less than a number of operation commands compatible as inputs to a current operational state of the application;responsive to determining that the number of determined states is less than the number of operation commands of the application, selecting (906) a subset of the operation commands of the application; andassigning (908) each operation command of the subset to a different state of the OBJ.

23. A computer program product for an Extended Reality, XR, device comprising a controller, the computer program product comprising:a computer readable medium storing instructions executable by the controller of the XR device to perform operations to:detect a physical object, OBJ, in a field of view, FOV, of an image sensor;determine a number of states of the OBJ that satisfies a rule for a user to interact with the OBJ to provide operational command inputs to an application;determine whether the number of determined states is less than a number of operation commands compatible as inputs to a current operational state of the application;responsive to determining that the number of determined states is less than the number of operation commands of the application, select a subset of the operation commands of the application; andassign each operation command of the subset to a different state of the OBJ.

Images