Assigning user input interfaces for an extended reality device to stable mechanical equilibrium states of physical object(s) tracked by the extended reality device

Abstract

Operations by an XR device detect physical object(s) for use by a user to provide input to user input interfaces controlling display of content through a display. Operations identify a number of stable mechanical equilibrium states of the physical object(s). Responsive to determining the number of stable mechanical equilibrium states is different than a number of the user input interfaces that are to be assigned, operations determine assignments of individual ones of the user input interfaces to at least one of the stable mechanical equilibrium states of the physical object(s) based on a characteristic of the user input interface and / or a characteristic of the at least one of the stable mechanical equilibrium states. Operations track user manipulation of a manipulated one of the physical object(s) relative to one of the stable mechanical equilibrium states, and provide input to the user input interface assigned to the stable mechanical equilibrium state.

Description

Attorney Docket P112001WO01 / 9900-112001WO01ASSIGNING USER INPUT INTERFACES FOR AN EXTENDED REALITY DEVICE TO STABLE MECHANICAL EQUILIBRIUM STATES OF PHYSICAL OBJECT(S) TRACKED BY THE EXTENDED REALITY DEVICETECHNICAL FIELD

[0001] The present disclosure relates to an extended reality (XR) device, a software component arrangement for an XR device, computer-readable medium carrying computer instructions, and a method for use in an XR device for providing an improved extended reality interface which allows for using objects in the surroundings as input objects.BACKGROUND

[0002] Augmented reality (AR) is a technology that overlays digital information or computergenerated graphics onto the physical world. It involves adding virtual elements to the user's actual environment, typically viewed through a camera (Video see-through VST) or a transparent screen (Optical see-through XR), such as in a headset, a full-body suit or through a smartphone. AR can enhance the perception of reality by adding layers of digital information that can be informative, entertaining, or both, or neither.

[0003] Virtual reality (VR), on the other hand, creates a completely immersive, computer-generated environment that can be experienced through a headset or a full-body suit. In VR, users are completely isolated from the real world and transported to a simulated environment that can be interactive and often responsive to user input.

[0004] The key difference between AR and VR is that AR enhances the physical world by overlaying digital elements onto it, whereas VR replaces the physical world with a computer-generated environment. AR appears therefore to be a more natural and intuitive experience that blends the virtual with the real world, while VR is a more immersive but completely synthetic experience.

[0005] Both AR and VR have a wide range of applications, including gaming, education, advertising, training, and simulation, among others.

[0006] Mixed reality (MR) is a term used to describe the merging of a real-world environment and a computer-generated one. Physical and virtual objects may co-exist in mixed reality environments and interact in real time.Attorney Docket P112001WO01 / 9900-112001WO01

[0007] Extended reality (XR) is a catch-all term to refer to augmented reality (AR), virtual reality (VR), and mixed reality (MR).

[0008] Head- worn devices, such as optical-see-through, XR, devices, for example XR glasses, are used more and more in AR and XR systems, and are thus becoming more and more commonplace.

[0009] When navigating an extended reality environment, such as in an augmented reality, the user may use specific gloves (or other devices) to give input or other control to the system. Such devices require that the user brings them along and that they are kept ready at hand to be used. Alternatively, the user may use gestures to give input or other control. Such gestures may be difficult to remember, especially since they may differ between applications and also, if there are many applications having different commands or options, the resulting number of gestures a user has to learn becomes quite high.

[0010] The patent document published as US2022101002A1 discloses a method of generating a virtual reality (" VR") experience includes recognizing a real-world (" RW") object in possession of a user of the VR experience and defining a VR object that corresponds to the RW object, wherein the VR object is configured to be used during the VR experience. The method also includes tracking the RW object to identify actions related to with the RW object and rendering the VR object in the VR experience to correspond with the actions related to the RW object. This document discusses a highly advanced system based on artificial intelligence learning how a user handles real-world objects. One example is where a user is able to handle a smartphone also in the virtual reality and where for example an image taken in the virtual reality is stored in the smartphone in the real-world.

[0011] The patent document published as US2021031110A1 discloses a system for generating video game inputs is provided. The system comprises an input unit operable to obtain images of a passive non-luminous object being held by a user as a video games controller. The system also comprises a physical object detector and object pose detector for detecting the physical object and its respective pose in the obtained images. The pose detector is configured to detect the pose of the physical object based on at least one of a (i) contour detection operation and (ii) the output of a machine learning model that has been trained to detect the poses of passive non-luminous objects in images. A user input generator is configured to generate user inputs based on the detected changes in pose of the passive non-luminous object and to transmit these to a video game unit at which a video game is being executed. A corresponding method is also provided. In this document any object pickedAttorney Docket P112001WO01 / 9900-112001WO01up by the user may be used as a game controller and is thus restricted to objects already picked up and selected by the user.SUMMARY

[0012] Various embodiments of the present disclosure are directed to improving how users can interact with physical objects in their environment to control operation of an XR device.

[0013] In some embodiments, an XR device includes a display, an image sensor, and a controller. The image sensor is configured to generate video imaging a field of view (FOV) of the image sensor. The controller is operative to detect at least one physical object imaged in the video from the image sensor to be analyzed for use by a user to provide input to user input interfaces controlling display of content through the display. The controller identifies a number of stable mechanical equilibrium states of the at least one physical object. Responsive to determining the number of stable mechanical equilibrium states is different than a number of the user input interfaces that are to be assigned, the controller determines assignments of individual ones of the user input interfaces to at least one of the stable mechanical equilibrium states of the at least one physical object based on a characteristic of the user input interface and / or a characteristic of the at least one of the stable mechanical equilibrium states. The controller tracks, in the video from the image sensor, user manipulation of a manipulated one of the at least one physical object relative to one of the at least one of the stable mechanical equilibrium states of the manipulated one of the least one physical object. The controller provides input to the user input interface assigned to the one of the at least one of the stable mechanical equilibrium states of the manipulated one of the at least one physical object, based on the tracked user manipulation.

[0014] In some further embodiments, the input is provided, e.g., as a command, to an user input interface of an operating system program of the XR device and / or to an application program (e.g., application programming interface) executed by the XR device, e.g., to start, stop, or control operation of an application program or OS program.

[0015] Because the user input interfaces are assigned to stable mechanical equilibrium states of the physical object(s), a user is able to more naturally interact with the physical object(s), e.g., rotate and / or translationally move the object relative to its stable mechanical equilibrium states, to more reliably control operation of an XR device.

[0016] Other XR devices and corresponding methods and computer program products according to embodiments of the present disclosure will be or become apparent to one with skill in the art uponAttorney Docket P112001WO01 / 9900-112001WO01review of the following drawings and detailed description. Moreover, it is intended that all embodiments disclosed herein can be implemented separately or combined in any way and / or combination.BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Embodiments of the invention will be described in the following, reference being made to the appended drawings which illustrate non-limiting examples of how the inventive concept can be reduced into practice.

[0018] Figure 1A shows a schematic view of electronic components that can be integrated into or operationally connected to an XR device according to some embodiments of the present disclosure,

[0019] Figure IB shows a schematic view of an XR device according to some embodiments of the present disclosure,

[0020] Figures 2A, 2B and 2C each shows a schematic view of a physical object and possible stable mechanical equilibrium states for such a physical object,

[0021] Figures 3A to 3H each shows a schematic view of an XR device according to some embodiments of the present disclosure,

[0022] Figures 4A to 4B each shows a schematic view of an XR device according to some embodiments of the present disclosure,

[0023] Figure 5 shows a flowchart of operations by the XR device according to some embodiments of the present disclosure,

[0024] Figure 6 shows a component view for a software component arrangement according to some embodiments of the present disclosure, and

[0025] Figure 7 shows a flowchart of operations by the XR device for handling three scenarios relating to comparison of a number of user input interfaces and a number of stable equilibrium states of an available physical object(s) that are to be manipulated by a user to provide input to an operating system program and / or an application program of the XR device, according to some embodiments of the present disclosure.DETAILED DESCRIPTION

[0026] Inventive concepts will now be described more fully hereinafter with reference to the accompanying drawings, in which examples of embodiments of inventive concepts are shown. InventiveAttorney Docket P112001WO01 / 9900-112001WO01concepts may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of various present inventive concepts to those skilled in the art. It should also be noted that these embodiments are not mutually exclusive. Components from one embodiment may be tacitly assumed to be present or used in another embodiment.

[0027] Figure 1A shows a schematic view of electronic components that can be integrated into or operationally connected to an XR device 100 according to some embodiments of the present disclosure, Hereafter both will be referred to as the XR device 100. The XR device 100 comprises a controller 101 and a memory 102. The XR device 100 also comprises or is connected to an image sensor 103. The XR device 100 also comprises or is connected to a display device 104. The image sensor 103 and / or the display device 104 are optional in that one or both of them may be connected to the XR device 100, and are thus considered to be comprised in the XR device 100 through the connection.

[0028] The controller 101 is configured to control the overall operation of the XR device 100. In some embodiments, the controller 101 may be a general-purpose controller, e.g., general purpose processor, which can perform a variety of tasks. As a skilled person would understand there are many alternatives for how to implement a controller, such as using Field -Programmable Gate Arrays circuits, ASICs, GPUs, etc. in addition to or as an alternative. For the purpose of this application, all such possibilities and alternatives will be referred to simply as the controller 101.

[0029] It is also noted that in some embodiments, parts of or all of the processing of the controller may be performed remotely, where a local controller 101 is configured to provide input data to a remote processing platform, such as to a cloud server or an edge network node, causing the remote processing platform to perform the processing and transmit the results of such processing as output to the local controller 101. For the purpose of this application, such possibilities and alternatives will also be referred to simply as the controller 101. The controller 101 may thus represent both the local controller and the remote processing platform.

[0030] The memory 102 is configured to store map data, graphics data, User Interface (UI) settings and computer-readable instructions (e.g., operating system instructions, application program instructions, etc.) that when loaded into the controller 101 indicate how the XR device 100 is to be controlled. The memory 102 may comprise one or more discrete memory circuits or hierarchy of memories, which are collectively referred to as being part of the same overall memory 102. There mayAttorney Docket P112001WO01 / 9900-112001WO01be one memory used by the display device storing graphics data, another memory used by the sensor for storing settings, another memory used by the communications interface (if such is present) for storing settings, and so on. As a skilled person would understand, there are many possibilities of how to select where data should be stored. In one embodiment, the XR device may comprise a general memory 102 in turn comprising any and all such memory circuits for the purpose of this application. As a skilled person would understand there are many alternatives of how to implement a memory, for example using non-volatile memory circuits, such as EEPROM memory circuits, or using volatile memory circuits, such as RAM memory circuits. For the purpose of this application all such alternatives will be referred to simply as the memory 102.

[0031] The display device 104 may in some embodiments be a display arrangement comprising one or more displays arranged to present visual data, predominantly through images. In some such embodiments, the display device 104 may be a touch screen thereby enabling user input to be provided to and received by the XR device 100. The visual data is related to the user interface of the XR device being presented by the XR viewing device 100. The XR device is thereby arranged to present image data through a (graphical) user interface in a manner controlled by the controller 101. When implemented as a head-worn XR device, the display device 104 may be configured as an optical see-through (OST) screen on which the user views video of the real-world captured by the image sensor 103, or may be configured as a see-through screen which allows the user to view the light from the real-world passed through the display screen with overlaid computer-generated graphics providing an augmented reality experience.

[0032] The image sensor 103, such as a camera or image sensor module, is arranged to provide an image (or stream of images) of the user environment when the user is utilizing the XR device 100, wherein images of the user environment may be analyzed using image processing techniques known in the art in order to determine or detect one or several objects in the surroundings. As a skilled person would understand, the XR device 100 may comprise one controller 101 and the sensor 103 may comprise another controller, but for the purpose of the teachings herein, they will be considered to be the same controller 101 in order to cover all possible variations of exactly where the determination of a physical object takes place.

[0033] Figure IB shows a schematic illustration of an XR device 100 in the form of a head-worn XR device in which the image sensor 103 provides a video stream capturing the real-world to the controller 101 for display on the display device 104. The controller 101 can operate to detect a physical object OBJ in the video stream from the image sensor 103.Attorney Docket P112001WO01 / 9900-112001WO01

[0034] The XR device 100 may in some embodiments be at least partially implemented in a smartphone. The smartphone may be carried in a head-worn carrier, enabling the smartphone to be used as an XR headset.

[0035] In some embodiments the XR device 100 is an AR or XR headset (also referred to as goggles).

[0036] As mentioned above, the inventors have realized that by determining stable mechanical equilibrium states of objects in the surroundings (as perceived by the image sensor 103) and to assign commands to these stable mechanical equilibrium states, objects that the user perceives may be manipulated by the user to provide input (e.g., program control) for giving commands to the XR device 100, e.g., as user input provided to an operating system (OS) program and / or to an application program executing in the XR environment.

[0037] A stable mechanical equilibrium state as referred to herein is to be understood to be a mechanical equilibrium state. In classical mechanics, a particle is in mechanical equilibrium if the net force on that particle is zero. By extension, a physical system made up of many parts is in mechanical equilibrium if the net force on each of its individual parts is zero. A stable mechanical equilibrium state is thus a state where the physical object will come to rest, unless subjected to external forces.

[0038] The inventors have realized that in most cases, a user's physical environment may not have many regular-shaped objects that are easy to manipulate, but that even irregularly shaped objects may be used, by determining a encompassing shape of the physical object.

[0039] The controller 101 can operate to determine the stable mechanical equilibrium states based on detecting and recognizing a physical object and then determining an encompassing shape of the physical object. Then the stable mechanical equilibrium states of the encompassing shape are determined and used as the stable mechanical equilibrium states of the physical objects. Figure 2A shows an example of a physical object (OBJ) having an irregular shape, in this example two triangles joined at their tops or edges. Figure 2A also shows an encompassing shape 105 having been determined for the physical object OBJ.

[0040] In some embodiments, the stable mechanical equilibrium states of a physical object (or the stable mechanical equilibrium states of the encompassing shape for a physical object) are determined as being the states, when the physical object (or encompassing shape thereof) is resting on an equilibrium plane. An equilibrium plane of a physical object - i.e., a plane on which the physical object is laying under gravity - is defined by the normal plane direction n. In order to find the equilibrium planes of aAttorney Docket P112001WO01 / 9900-112001WO01physical object (such as exemplified by a 3D-model), the (convex-)encompassing shape and the center of mass of the physical object first need to be computed.

[0041] It is noted that in the description below, reference is made to a 3D model of a physical object. However, these references should be assumed to be also to the physical object as detected by the imaging sensor. The 3D model may thus also be referenced as a 2D model, being a specific nonlimiting example of a 3D model.

[0042] The convex encompassing shape (or hull) of a physical object is the smallest convex polyhedron containing the physical object the physical object. It is a fundamental construction for mathematics and computational geometry and will therefore not be discussed in great detail as a skilled person can find details on this through various known articles and textbooks. Its construction can be achieved in many ways. Some operations that may be used by the controller 101 are based on deterministic incremental algorithms or randomized incremental construction. In some embodiments a CGAL implementation of the QuickHull algorithm is utilized. It handles all degenerate cases and nonmanifold models of objects that constitute 3D-model database. In the worst case this algorithm is O(n2), but in practice it is no worse than O(n*log(n)), where n is the number of vertices of a 3D-model.

[0043] The calculation of the center of mass and the volume of rigid bodies has also been extensively treated in literature. One efficient approach that can be used by the controller 101 to compute the center of mass for polyhedral objects will be discussed, but it should not be that there are many other alternative algorithms available. This algorithm is based upon a three-step reduction of the volume integrals to successively simpler integrals. The final step of the algorithm computes the required integrals over a face from the coordinates of the projected vertices. Considering that mass distribution is homogeneous in the 3D-model volume V, the center of mass G thus yields:1 r r 1” |. i

[0044] An orthogonal projection along the z direction thus results in a more convenient and efficient integral, as follows:Attorney Docket P112001WO01 / 9900-112001WO01

[0045] where S is the surface boundary of the 3D-model, n is the normal at point P(x, y, z) and sign(x) denotes the signum function which extracts the sign of a real number x.

[0046] Using the divergence theorem, the 3D-model volume V yields:

[0047] The overall complexity of the two integrals is O(m), where m represents the number of faces of the polyhedron.

[0048] Based on this the equilibrium planes may be computed by assuming the following conditions to compute an equilibrium plane:

[0049] Theorem 1 A direction n defines an equilibrium plane E if and only if there exists a plane n of normal n such that it contains a face Fi of the convex encompassing shape H of the 3D-model / object (convex encompassing shape condition) and if the projection of the center of mass G of the 3D-model / object along n is inside Fi (center of mass condition).

[0050] Proof A static mechanical equilibrium requires that at least three non collinear points of the 3D-model belong to n. Since n is not a separating plane of the 3D-model points, it necessarily contains a face Fi of the convex encompassing shape H. Let us suppose the projection of G along n onto n does not belong to Fi. The gravity force applied on G thus exerts a moment whose axis A is the nearest edge of Fi from G. The ground force also exerts a moment on A, that does not oppose to the gravity moment. So their resultant moment on A is not zero and the 3D-model is thus not in static equilibrium. The projection of G along n onto n must consequently belong to Fi.This provides the equilibrium plane computation:Algorithm 1 Equilibrium plunc idp rilhmCompute the \i. Mi l hullCompute the (.cuter of r. Ki^ t >.for all Ibccs ot ". Jf dolJrn|cct orthummJIv h mt ph.ue r containing J-f’F thenAdd π direction n to the equilibrium plane list.end ifend for

[0051] The Computational Geometry Algorithms Library, CGAL, comprises many algorithms that are available to a skilled person for implementing such determinations as herein. The CGAL is a softwareAttorney Docket P112001WO01 / 9900-112001WO01project that provides easy access to efficient and reliable geometric algorithms in the form of a C++ library..

[0052] As noted above, stable mechanical equilibrium states may thus be determined for any object in the user's environment and these stable mechanical equilibrium states may be assigned commands. A user may thus be enabled to input commands or other control by manipulating these objects in the surroundings.

[0053] Figure 2B shows a schematic illustration of how a encompassing shape 105 can be placed in different positions or be placed in different stable mechanical equilibrium states. The encompassing shape 105 is the same as that of Figure 2A, and it can assume three stable mechanical equilibrium states (in two dimensions). As mentioned above, the teachings herein are discussed with relation to two dimensions, but they equally apply to three dimensions. In Figure 2B three states are shown; standing, laying and standing up-side-down. Depending on the shape of the physical object OBJ, there may be different number of states for the laying state in Figure 2B. Figure 2C shows examples of the number of different states depending on the shape of the encompassing shape 105 of the physical object in Figure 2A. For illustrative purposes, four different examples are shown where the encompassing shape has 3, 4, 5 or 6 different stable mechanical equilibrium states (when laying).

[0054] The general operation of an XR device 100 according to some embodiments, and specifically as illustrated in Figures 1A and / or IB, will now be discussed with reference to Figures 3A to 3H. In the following discussion operations are described in the context of a user manipulating a physical object OBJ to provide user input to the OS and / or application program operating in the XR device 100. The user input is also referred to as a command that input by the user, whereby different user input can correspond to different commands being input by the user. The term "command" below can therefore be replaced with the more general term "user input" or "input".

[0055] Figure 3A shows a schematic view of an XR device 100 that detects a physical object OBJ in the surroundings utilizing the image sensor 103 and determines an encompassing shape 105 for the physical object OBJ (Figure 3B) and the stable mechanical equilibrium states for the encompassing shape 105. The XR device 100 also assigns commands to at least some of the states of the encompassing shape 105 (after determining that the physical object is a usable physical object). In order to enable the user to manipulate the physical object correctly, the XR device 100 may, in some embodiments, display the assigned commands on the display device 104, as virtual labels or other graphic indicators 104-R, Figure 3C.Attorney Docket P112001WO01 / 9900-112001WO01

[0056] The XR device then detects a manipulation of the usable physical object OBJ, Figure 3D. The manipulation may be detected by tracking the hands of the user, or simply by detecting a change in the placement of the usable physical object OBJ. As the usable physical object has been detected to have been manipulated the XR device 100 determines which stable mechanical equilibrium state that the usable physical object (or the encompassing shape thereof) has assumed or been placed in, and then executes the command that is assigned to that stable mechanical equilibrium state, Figure 3E.

[0057] The teachings herein thus provide an XR device 100 configured to, detect a usable physical object OBJ in the field of view FOV of the user (i.e. the surroundings as perceived through video output by the image sensor 103), where the usable physical object is defined as a physical object having two or more stable mechanical equilibrium states, determine two or more stable mechanical equilibrium states of the usable physical object, assign the one or more operating commands of the application to the stable mechanical equilibrium states of the usable physical object, detect a user manipulation of the usable physical OBJ, determine a resulting stable mechanical equilibrium state of the usable physical object after user manipulation, and execute the command assigned to the resulting stable mechanical equilibrium state.

[0058] It is noted that the example of Figures 3A to 3E show the usable physical object being manipulated from a first placement / state to a second state. However, the association of commands may also be made for a series of manipulations where the command is not only associated to the resting or final state, but to a defined series of intermediate state(s). A first command may thus be associated with placing the usable physical object directly in a first stable mechanical equilibrium state (for example turning up-side-down). A second command may thus be associated with placing the usable physical object in the second stable mechanical equilibrium state (up-side-down), but by first placing it in an intermediate state (laying). The teachings herein thus provide, in some embodiments, an XR device 100 further configured to determine one or more intermediate stable mechanical equilibrium states assumed during the user manipulation of the usable physical object, and wherein the command assigned to resulting stable mechanical equilibrium state depends on the one or more intermediate stable mechanical equilibrium states assumed during the user manipulation.

[0059] As the investors have realized, not all physical objects are usable for manipulation. The teachings herein thus also provide for detecting only usable physical objects.

[0060] A physical object may be usable if it has enough states to represent the commands of the current XR environment / application. If a physical object only has two states, it may be deemed to not beAttorney Docket P112001WO01 / 9900-112001WO01usable for an application having five commands. The teachings herein thus provide, in some embodiments, an XR device 100 further configured to determine a required number of operating commands of the application and to detect a physical object as a usable physical object if the number of stable mechanical equilibrium states of the physical object is equal to or higher than the required number of operating commands.

[0061] If a physical object is deemed to not be suitable, the XR device 100 is determined, in some embodiments, to detect a second physical object and determine if the second object may be a usable object, Figure 3F. The teachings herein thus provide, in some embodiments, an XR device 100 further configured to determine another physical object if the number of stable mechanical equilibrium states of the first object is less than the required number of command operations.

[0062] As the inventors have realized, a physical object having more stable mechanical equilibrium states than the commands of the XR environment / application may still be utilized, and a command may be associated with more than one stable mechanical equilibrium state. The teachings herein thus provide, in some embodiments, an XR device 100 further configured to assign at least one of the operating commands to multiple stable mechanical equilibrium states when assigning operating commands to stable mechanical equilibrium states, when the number of stable mechanical equilibrium states is higher than the number of operating commands.

[0063] Specifically, the teachings herein thus provide, in some embodiments, an XR device 100 further configured to detect a physical object as a useful physical object if the number of stable mechanical equilibrium states is equal to the number of operating commands.

[0064] As the inventors have also realized, a physical object may still be usable - even if the number of stable mechanical equilibrium states is less than the number of commands - by assigning commands to sequences of stable mechanical equilibrium states (as discussed above with reference to intermediate stable mechanical equilibrium states).

[0065] The inventors have also realized that even if an encompassing shape or shape of a physical object may have many stable mechanical equilibrium states, not all these states are possible for a physical object. For example, a cup may not be possible to place in all states of the corresponding encompassing shape, such as up-side-down. The teachings herein thus provide, in some embodiments, an XR device 100 is further configured to determine a number of possible stable mechanical equilibrium states that the possible object can assume and set the number of possible states that can be assumed as the number of stable mechanical equilibrium states for the physical object.Attorney Docket P112001WO01 / 9900-112001WO01

[0066] In some embodiments the XR device is configured with (such as being stored in the memory 102 or being retrievable from a networked computer server) a library of stored physical objects and associated possible encompassing shapes. A physical object that is detected can be compared to the library of encompassing shapes in order to determine the number of possible stable mechanical equilibrium states. The teachings herein thus provide, in some embodiments, an XR device 100 that is further configured to determine a number of possible stable mechanical equilibrium states that the possible object can assume by matching the encompassing shape to stored shapes, wherein the stored shapes have unallowable stable mechanical equilibrium states.

[0067] Alternatively or additionally, the detected physical object may be classified or otherwise identified through a physical object segmentation. Such object segmentations are known in the art of image processing and will not be further discussed herein. The physical object segmentation may thus identify a physical object as a mug, which has an unallowable state being up-side-down associated with it. The teachings herein thus provide, in some embodiments, an XR device 100 further configured to determine a number of possible stable mechanical equilibrium states that the possible physical object can assume by utilizing object segmentation to identify a type of physical object, wherein some types of physical objects have unallowable stable mechanical equilibrium states.

[0068] Alternatively or additionally, the XR device may further incorporate a thermal imaging sensor (assumed to be part of the imaging sensor 103), whereby a physical object may be deemed unusable based on a temperature of the physical object. For example, a physical object that has a high temperature may be determined to be unsuitable for manipulation. Similarly, a physical object may be determined as unsuitable if it is too cold. The teachings herein thus provide, in some embodiments, an XR device 100 further configured to determine a number of possible stable mechanical equilibrium states that the possible physical object can assume by determining that the physical object has a temperature that is above, below or at a temperature threshold level, and wherein the controller is configured to in that case set the number of possible stable mechanical equilibrium states for the possible object to zero.

[0069] Alternatively or additionally, the XR device may further incorporate an inertial sensor (assumed to be part of the sensor 103), whereby the XR device can determine that it is in a moving (or shaking) environment such as on a bus or a train, wherein a number of states may not be possible for a encompassing shape. Such as when a state requires only small forces to be toppled. The teachings herein thus provide, in some embodiments, an XR device 100 further configured to determine a numberAttorney Docket P112001WO01 / 9900-112001WO01of possible stable mechanical equilibrium states that the possible physical object can assume by determining that the inertial sensor (IMU) 103 indicates a movement level that exceeds a movement threshold level, and wherein the controller is configured to in that case set the number of possible stable mechanical equilibrium states for the possible physical object to zero.

[0070] If the XR device is not able to detect any usable physical object in the current environment (field of view), the XR device 100 may be configured in some embodiments to prompt the user to change the environment. In some embodiments the environment may be changed by moving (laterally and / or rotationally) so that a new environment moves into the field of view, whereby new usable physical objects will come into view. In some embodiments the environment may be changed by moving physical objects around in the current view, whereby new objects may come into view, such when certain physical objects not usable for any action in the XR environment or XR applications partially or entirely obscure usable physical objects. Figures 3G and 3H show an example where two objects may be identified as a single object (Figure 3G) not having many usable states, but when moved around (figure 3H) actually reveals two objects each having usable states. The teachings herein thus provide, in some embodiments, an XR device 100 further configured to indicate to the user to change environment when no usable physical object can be detected.

[0071] Alternatively or additionally, the XR device is configured to enable a user to select which usable physical object to be used for manipulation. In some such embodiments the XR device is configured to indicate usable physical objects to a user, by for example marking them on the display 104, and then receiving an indication from the user as to which usable physical object should have motion and / or location tracked for providing user input. For example, the user may be presented with graphical markings indicating two usable physical objects, and then select one of them by pointing at the usable physical object selected. The imaging sensor 103 will be able to track the hand and detect the pointing, whereby the pointed to usable physical object will be identified by the controller 101. The teachings herein thus provide, in some embodiments, an XR device 100 further configured to select the usable physical object by indicating one or more usable physical objects to the user, receiving user indication of one usable physical object and selecting the indicated usable physical object for controlling the XR environment or the XR application currently running.

[0072] As discussed in the above, the commands associated with a usable physical object may be indicated graphically to the user. As is shown in Figures 4A and 4B, the commands associated with a usable physical object OBJ may be indicated through a graphical representation 104-R on the displayAttorney Docket P112001WO01 / 9900-112001WO01104, wherein a graphical indication 105R of the usable physical object may be shown, such as a graphical indication of the encompassing shape of the usable physical object. A label or other graphical indication of a command 104C may be shown on or adjacent to the side of a state associated with the command. In Figure 4A there are two commands COM1, COM2 indicated through graphical indications 104C1, 104C2 along the associated sides.

[0073] As a usable physical object is manipulated the XR device may be arranged to indicate that the associated command is being (or to be) executed. This enables a user to understand the consequence of the manipulation. In some embodiments (general) confirmation is required to execute the command. The indication is in some embodiments provided by changing or adapting the graphical indication of the command. In Figure 4B this is shown as the graphical indication 104C2 for the second command COM2 having been changed as the physical object has been manipulated (placed in a laying position).

[0074] The teachings herein thus provide, in some embodiments, an XR device 100 further configured to indicate to the user at least one assigned operating command to a stable mechanical equilibrium state of the encompassing shape.

[0075] As noted above, the stable mechanical equilibrium states for a usable physical object are determined based on the encompassing shape of the usable physical object in the form of a (convex) hull. The XR device 100 is thus, in some embodiments, configured to determine an encompassing shape for the physical object OBJ by determining a convex hull encompassing the usable physical object OBJ.

[0076] However, other manners of determining a shape encompassing the usable physical object may be used, and the teachings herein relating to a hull, thus also apply to any other shape used. In some embodiments the XR device 100 is further configured to determine an encompassing shape for the usable physical object OBJ by determining a shape for the detected usable physical object and matching the shape of the detected usable physical object to known shapes stored in the memory 102.

[0077] Figure 5 shows a general flowchart for various operations by the XR device 100 as discussed above. Although operations are illustrated in Figure 5 as being performed in a sequence, it is to be understood that the order of operations can be changed without restriction, that two or more operations that are shown in sequence may be performed in parallel, and that two or more of the operations may be performed repetitively more often than other illustrated operations.

[0078] The operations include determining 510 one or more operating commands or other user input interfaces applicable to an XR environment or an application running in the XR environment,Attorney Docket P112001WO01 / 9900-112001WO01detecting 520 a usable physical object (OBJ) in the FOV of the user, determining 530 an encompassing shape for the usable physical object (OBJ), wherein the encompassing shape is a geometrical shape that encompasses the usable physical object (OBJ). The operations further include determining 540 two or more stable mechanical equilibrium states of the encompassing shape, selecting 550 the usable physical object as a command object (or other user input interface object) to be used. The command (user interface object) object can be a usable physical object seen by the user through the XR device and when manipulated by the user indicates a command (input to a user interface) to be performed by the XR device, and assigning 560 each of the one or more operating commands (user input interfaces) of the application to each stable mechanical equilibrium state of the encompassing shape for the command object. The operations further include detecting 570 a user manipulation of the command object (OBJ), determining 580 a resulting stable mechanical equilibrium state of the command object after user manipulation and executing 590 the command (user input interface) assigned to the resulting stable mechanical equilibrium state.

[0079] Figure 6 shows a component view for a software component or module arrangement 600 according to some embodiments of the teachings herein. The software component arrangement 600 is adapted to be used in an XR device 100 as taught herein for providing adaptation as taught herein and corresponds to the operation of the XR device 100 in the above. The software component arrangement 600 comprises a software component 610 for a software component for determining one or more operating commands (user input interfaces) that can be into to an OS program of an XR environment or an application program running in the XR environment, a software component 620 for detecting a usable physical object (OBJ) in the FOV of the user, a software component for determining 630 an encompassing shape for the usable physical object (OBJ), wherein the encompassing shape is a geometrical shape that encompasses the usable physical object (OBJ). The software component arrangement 600 further comprises a software component 640 for determining two or more stable mechanical equilibrium states of the encompassing shape, a software component 650 for selecting the usable physical object as a command object (user input interface object) to be used, where the command object (user interface object) can be a physical object (e.g., user selectable menu item, indicia, etc.) seen by the user through the XR device and when manipulated by the user indicates a command (input to a user input interface) to be performed by the XR device, and a software component for 660 assigning each of the one or more operating commands (inputs to user input interfaces) of the OS and / or application to each stable mechanical equilibrium state of the encompassing shape for theAttorney Docket P112001WO01 / 9900-112001WO01command object. The software component arrangement 600 further comprises a software component 670 for detecting a user manipulation of the command object (OBJ), a software component 680 for determining a resulting stable mechanical equilibrium state of the command object after user manipulation and a software component 690 for executing the command (input to user input interface) assigned to the resulting stable mechanical equilibrium state. The software component arrangement 600 may further comprise software component(s) 695 for further functionality as discussed herein.

[0080] Various aspects of the XR device described above assume that the operations are able to select a physical object that has a number of stable mechanical equilibrium states which is preferably equal to the number of user input interfaces (also called "control states" and "required states (RS)") of the operating system (OS) program and / or application program of the XR device that are to be made available for a user to provide input to through manipulation of the physical object. Some of these embodiments address a situation where the number of stable mechanical equilibrium states determined for an available physical object resembles, e.g. is the same as the number of required number of user input interfaces, referred to as a "perfect-fit scenario."

[0081] It is, however, expected that a general real-world environment, e.g. office desk, will not be occupied with and provide the XR user with physical objects that have a number of stable mechanical equilibrium states that matches the number of user input interfaces which need to be assigned. For example, the XR device operations may be adapted to assign a defined number of user selectable menu items to a same number of stable mechanical equilibrium states of a single physical object. However, the operations may be unable to identify a single physical object which is determined to have that number of stable mechanical equilibrium states.

[0082] In an illustrative operational example, an XR application has four menu options which are to be displayed via display 104 to the user and are supposed to be mapped to four stable mechanical equilibrium states (also called "stable equilibrium states" for brevity) of a physical object which is present in the video from the image sensor 103. The user's desk has only four objects which are observed in the video: a ball, a triangle-based pyramid (tetrahedron) object, a smooth AirPods case (or something with that formfactor), and Rubrik's cube. In this example, the operations cannot select the ball because it is determined to not have the required number of stable equilibrium states. The operations should also not select the AirPods case because because it is determined to have only two stable equilibrium states (top and bottom surfaces of the case) and would therefore not be suitable since only two of the four XR menus can be assigned to stable equilibrium states. In contrast, theAttorney Docket P112001WO01 / 9900-112001WO01operations can select the pyramid (tetrahedron) object because it can assign each of the four XR menu options to a different one of the four stable equilibrium states of the object, i.e., a one-to-one mapping between menu options and stable equilibrium states. However, there are typically a plurality of physical objects on a desk or other surface and which have other numbers of possible stable states than the exact number of of user input interfaces which are to mapped for user input.

[0083] Various embodiments of the present disclosure are directed to operationally enabling the XR device to assign a determined number of user input interfaces of an OS and / or application program to a greater number of stable equilibrium states of a single physical object (referred to as "overfitted scenario") or to a select a set of physical objects which individually do not have a sufficient number of stable equilibrium states (referred to as "underfitted scenario") but that collectively provide the same or greater number of stable equilibrium states as the number of user input interfaces.

[0084] In the overfitted scenario, the number of stable equilibrium states of an available physical object is larger than the number of user input interfaces, e.g., XR-application menu entries, and the problem to solve here is how the operations shall assign (also called "map") the fewer number of user input interfaces to more than enough available number of stable equilibrium states for the corresponding physical object. For example, the operations should solve how to assign A menu entries among an object's B stable equilibrium states (where B> A), i.e., where not all of the stable equilibrium states B of a single object need to be assigned one of the A user input interfaces.

[0085] In the underfitted scenario, the number of stable equilibrium states of an available individual physical object is smaller than the number of user input interfaces, e.g., XR-application menu entries, and the problem to solve here is how the operations shall select a set of physical objects that are to be used as a combination that provides a combined number of stable equilibrium states that is greater than the required number of user input interfaces. For example, the operations should solve how to assign A menu entries among a set of objects having a combined number B stable equilibrium states, where no individual one of those objects has a number of stable equilibrium states that is equal to or greater than A. In the underfitted scenario, the mapping cannot be obtained using only one physical object but more than one physical object can be selected for use as a set which collectively provide the required number of menu entries.

[0086] In these operational embodiments, a physical objects' stable equilibrium states can be assigned to user input interfaces of the OS and / or application, such that a user can manipulate the physical object relative to those stable equilibrium states to provide commands and other input. ForAttorney Docket P112001WO01 / 9900-112001WO01example, an ordinary cube has six stable equilibrium states, but another physical object with certain convex surfaces may not be stable in every one of it possible stable equilibrium states. For example, a smartphone embedded in a protective case is typically stable while resting on its face and back surfaces on a tabletop, and although may be delicate balanced to stand up when positioned on one of its edge surfaces, the edge surfaces may not provide a sufficiently stable equilibrium state position for the smartphone to rest and for the operations to reliably assign a user input interface that stand up position.

[0087] Some further embodiments are directed to operations that can assign (map) at least one physical object's stable equilibrium state to a user input interface, e.g., an XR application user selectable menu item, when the physical object's number of equilibrium states are overfitting (e.g., number of stable equilibrium states > number of user input interfaces) or underfitting (e.g., number of stable equilibrium states < number of user input interfaces) with respect to the number of number of user input interfaces (e.g., XR application user selectable menu items) that need to be assigned.

[0088] Operational embodiments are provided which determine a physical object's stable equilibrium states and assign (map) those states to selected ones of the user input interfaces of the XR OS and / or application, e.g., the user selectable menu items, other indicia, etc. which are viewable by the user through the XR device.

[0089] Operations that can be performed by the controller 101 of the XR device 100 are now described with regard to the flowchart of operations shown in Figure 7. These operations can be used when the number of stable equilibrium states is equal to the number of user input interfaces that are to be assigned. Moreover, in accordance with further embodiments disclosed herein, these operations are adapted to be effectively used when there is an overfitted scenario or an underfitted scenario. Although operations are illustrated in Figure 7 as being performed in a sequence, it is to be understood that the order of operations can be changed without restriction, that two or more operations that are shown in sequence may be performed in parallel, and that two or more of the operations may be performed repetitively more often than other illustrated operations.

[0090] Referring to Figure 7, the operations detect 700 at least one physical object imaged in the video from the image sensor to be analyzed for use by a user to provide input to user input interfaces controlling display of content through the display. The operations identify 710 a number of stable mechanical equilibrium states of the at least one physical object. The operations determine 720Attorney Docket P112001WO01 / 9900-112001WO01whether the number of stable number of stable mechanical equilibrium states is different than a number of the user input interfaces that are to be assigned, the operations determine.

[0091] When the number of stable mechanical equilibrium states is determined 720 to be the same as the number of the user input interfaces that are to be assigned (perfect fit scenario), the XR device may perform the operations described above relating to operational steps 550-590 in Figure 5.

[0092] In contrast, when the number of stable mechanical equilibrium states is determined 720 to be the same as the number of the user input interfaces that are to be assigned, the operations determine 730 assignments of individual ones of the user input interfaces to at least one of the stable mechanical equilibrium states of the at least one physical object based on a characteristic of the user input interface and / or a characteristic of the at least one of the stable mechanical equilibrium states. The operation then track 740, in the video from the image sensor 103, user manipulation of a manipulated one of the at least one physical object relative to one of the at least one of the stable mechanical equilibrium states of the manipulated one of the least one physical object. The operations responsively provide 750 input to the user input interface assigned to the one of the at least one of the stable mechanical equilibrium states of the manipulated one of the at least one physical object, based on the tracked user manipulation.

[0093] The operation of the controller 101 of the XR device 100 (also referred to as "operation of the XR device 100" or "operation" for brevity) to identify 710 the number of stable mechanical equilibrium states of the at least one physical object, may include, for each of the at least one physical object, to: determine an encompassing shape for the physical object, wherein the encompassing shape comprises a plurality of geometrical shapes; and determine the number of stable mechanical equilibrium states of the physical object based on how many of the plurality of geometrical shapes have features which are determined to be capable of supporting the physical object in a stable mechanical equilibrium state while resting on a physical surface.

[0094] The operations may be further configured to determine the number of stable mechanical equilibrium states of the physical object based on which of the plurality of geometrical shapes have features matching features of stored geometrical shapes defined as capable of supporting a physical object in a stable mechanical equilibrium state while resting on a physical surface.

[0095] In this manner, the XR device 100 can operate to enable a user manipulate one or more physical objects to provide corresponding input to assigned user input interfaces of an OS program and / or application program of the XR device 100. The operations can handle all of the above-describedAttorney Docket P112001WO01 / 9900-112001WO01scenarios: perfect fit scenario, underfitted scenario, and overfitted scenario, to more reliably control operation of the XR device 100 under more diverse real-world situations where the user has the ability to use a wider variety of physical objects to provide input (e.g., control) to programs of the XR device 100.

[0096] These operational embodiments are first described in the context of the overfitted scenario and are then described in the context of the underfitted scenario.

[0097] Overfitting scenario:

[0098] As explained above, in the overfitted scenario the number of stable equilibrium states of an available physical object is larger than the number of user input interfaces, e.g., XR-application menu entries. Various embodiments are therefore directed to assigning the fewer number of user input interfaces (e.g., menu entries) to more than enough available number of user input interfaces for the corresponding physical object.

[0099] In one embodiment, the operations to detect 700 at least one physical object imaged in the video from the image sensor 103, can include to: determine a required number of the user input interfaces that are to be assigned; and select one physical object for manipulation by the user to provide input to the user input interfaces, based on determining the number of stable mechanical equilibrium states of the one physical object is at least as great as the required number of the user input interfaces.

[0100] The operations can be configured to assign one of the user input interfaces to more than one of the stable mechanical equilibrium states. The controller 101 of the XR device 100 can respond to determining 720 the number of stable mechanical equilibrium states is greater than the required number of the user input interfaces that are to be assigned, by operating to assign 730 one of the user input interfaces to more than one of the stable mechanical equilibrium states.

[0101] When assigning one of the user input interfaces to more than one of the stable mechanical equilibrium states, the operations may seek to assign the user input interface to a spatially adjacent set of stable mechanical equilibrium states. Therefore, in the context of Figure 7, the operations to assign 730 one of the user input interfaces to more than one of the stable mechanical equilibrium states can include to: identify a set of at least two stable mechanical equilibrium states of the one physical object that satisfy a rule for being spatially adjacent; and assign the one of the user input interfaces to the identified set of at least two stable mechanical equilibrium states.

[0102] Alternatively or additionally, when assigning one of the user input interfaces to more than one of the stable mechanical equilibrium states, the operations may seek to assign the user inputAttorney Docket P112001WO01 / 9900-112001WO01interface to a set of stable mechanical equilibrium states identified as requiring an amount of rotation and / or movement distance to transition between by user manipulation of the one physical object which satisfies a rule. Therefore, in the context of Figure 7, the operations to assign 730 one of the user input interfaces to more than one of the stable mechanical equilibrium states can include to: identify a set of at least two stable mechanical equilibrium states of the one physical object that satisfy a rule for an amount of rotation and / or movement distance required to transition between by user manipulation of the one physical object; and assign the one of the user input interfaces to the identified set of at least two stable mechanical equilibrium states.

[0103] The rule for the amount of rotation and / or movement distance required to transition between the at least two stable mechanical equilibrium states in the set by user manipulation of the one physical object, may configure the operations to perform a define one of maximizing or minimizing the amount of rotation and / or movement distance required to transition between the at least two stable mechanical equilibrium states in the set by user manipulation of the one physical object.

[0104] Alternatively or additionally, when assigning the user input interfaces to the stable mechanical equilibrium states, the operations may seek to space apart at least two of the assignments so that there is physical spacing between nearest ones of the stable mechanical equilibrium states assigned to different user input interfaces, e.g., to provide guard band spacing. Therefore, in the context of Figure 7, the controller 101 of the XR device 100 can respond to determining 720 the number of stable mechanical equilibrium states is greater than the required number of the user input interfaces that are to be assigned, by performing the determination 730 of the assignment of the user input interfaces to the stable mechanical equilibrium states while reserving at least one of the stable mechanical equilibrium states that is not assigned to any of the user input interfaces. The reserved at least one of the stable mechanical equilibrium states is selected by the operations to be positionally located on the one physical object between a spaced apart pair of two other stable mechanical equilibrium states that are mapped to two different user input interfaces.

[0105] Some further embodiments are directed to using relative probabilities that respective ones of the user input interfaces will be selected next for input by the user.

[0106] In the context of Figure 7, the controller 101 of the XR device 100 can respond to determining 720 the number of stable mechanical equilibrium states is greater than the required number of the user input interfaces that are to be assigned, by determining 730 assignments of the user input interfaces to the stable mechanical equilibrium states of the one physical object based onAttorney Docket P112001WO01 / 9900-112001WO01comparison of probabilities that respective ones of the user input interfaces will be selected next by the user to receive user input while the XR device is operating in a present operational state. The operations may, for each of the user input interfaces, compute the probability based on values indicating likelihood of the XR device transitioning from the present operational state to another allowed operational state by user input provided to the respective user input interface.

[0107] The operation of the controller 101 of the XR device 100 to determine assignments 730 of the user input interfaces to the stable mechanical equilibrium states of the one physical object, may include to assign one of the user input interfaces having the greatest probability to a greater number of the stable mechanical equilibrium states than another one of the user input interfaces have a lowest probability.

[0108] Thus, for example, when a set of user-selectable items are to be displayed on the display 104 of the XR device 100, the operations may be configured to assign 730 more, e.g., number X, stable mechanical equilibrium states of the one physical object (e.g., two states corresponding to physically adjacent surfaces, such as adjacent sides of a cube) to one of the user-selectable items that is determined to be most likely to be selected by the user based on prior user-selections of other items by the user and / or based on probabilities that have been defined for which operations states the user will want to transition to next from a present operational state of the OS and / or application program of the XR device 100. In contrast, the operations may be configured to assign 730 less, e.g., number Y where Y< X, stable mechanical equilibrium state(s) of the one physical object to another one of the user-selectable items that is determined to be less likely to be selected by the user based on prior userselections of other items by the user and / or based on probabilities that have been defined for which operations states the user will want to transition to next from a present operational state of the OS and / or application program of the XR device 100.

[0109] In further embodiment, the transition probabilities are inferred based on characteristics of the user input interfaces for the operating system program and / or application program, and may be based on Markov chain transition probabilities. The operations may, for example, determine:• %ratio Al of highest transition probability between a first and a second operational state of the program,• %ratio A2 of second highest transition probability between a first and a third operational state, • %ratio A3 of third highest transition probability between a first and a fourth operational state, etc.Attorney Docket P112001WO01 / 9900-112001WO01• %ratio Ai of i:th transition probability... etc.• % ratio A_remainder that collects other less used transition probabilities (where A_remainder > resolve_individual_states threshold. The operations can collect rarely used states into a separate group state and unfold that collection group state into a new loop, where respective Al, A2, A3... Ai [0..100%].

[0110] The operations can calculate a number of AS-allocations (nASa) that a highest transition probability Al corresponds to among control object AS, such as:• nASa_l = ceil(Al* AS) (... or alternatively nASal = floor(Al* AS), or "round")typically a floor-truncation will act conservatively and conserve %fraction-space allocations to allow more smaller fraction to perhaps get state allocations.• nASa_2 = ceil(A2* AS), nASa_3 = ceil(A3* AS),...• nAS _remainder= ceil(A_remainder* AS).

[0111] The operations can render (in the aspect of executing state-representing allocations to the control object), according to determined respective nASa_i, such as:• a number of states "nASa_l" are allocated to the control object representing the most probable transition probability from a starting state S.• a number of states "nASa_2" are allocated to the control object representing the most probable transition probability from a starting state S.• Etc.

[0112] The inclusion of " A_remainder" is intended to gather "other" less used transition probabilities associated with digital application interaction that are rather infrequently used, but nevertheless still possible.

[0113] Operations may compare to a hierarchical set of user input interfaces, e.g., menu items with some having a logical hierarchical association (e.g., menu-unfold) that are present in today's menu solutions to save space and improve most common user applications interactions so to unhide less / rarely / never used user input interfaces of application functions, etc., as the user interacts with certain application icon / objects and / or hovers mouse pointer over longer than some predefined / user-specific threshold.

[0114] The A_remainder can be determined so as to gather less frequently user states, and be assigned a certain control object accessible state. As the user interacts with the physical object's assigned A_remainder state, the operations responsively select, according to a defined rule (e.g., a topAttorney Docket P112001WO01 / 9900-112001WO01down, alphabetically, or otherwise predefined hierarchical association), the remainder of possible required stable mechanical states that are not yet subject to user's active transitions can be assigned to remaining user input interfaces.

[0115] Alternatively or additionally, when the operations determine 720 that the number of stable mechanical equilibrium states is greater than the required number of the user input interfaces that are to be assigned, the operations may determine 730 assignments of the user input interfaces to the stable mechanical equilibrium states of the one physical object based on comparison of relative sizes of physical object surfaces, which correspond to the determined mechanical equilibrium states of the physical object, to the sizes of user-selectable items that will be displayed to the user to represent the corresponding user input interfaces. Thus, for example, when a set of user-selectable items are to be displayed on the display 104 of the XR device 100, the operations may determine assignments of the user input interfaces to the stable mechanical equilibrium states of the one physical object based on comparison of sizes of physical object surfaces, which correspond to the determined mechanical equilibrium states, to the sizes of user-selectable items that will be displayed to the user to represent the corresponding user input interfaces.

[0116] Thus, for example, larger ones of the displayed menu items (or other indicia) can be mapped to larger ones of the physical object surfaces of the object, and smaller ones of the displayed menu items (or other indicia) can be mapped to smaller ones of the physical object surfaces of the object. In one embodiment, the largest displayed menu item (or other indicia) is mapped to the largest physical object surface of the object (which corresponds to one of the mechanical equilibrium states) and the smallest displayed menu item (or other indicia) is mapped to the smallest physical object surface of the object (which corresponds to another one of the mechanical equilibrium states).

[0117] Underfitting scenario:

[0118] As explained above, in the underfitting scenario the number of stable equilibrium states of an available physical object is smaller than the number of user input interfaces, e.g., XR-application menu entries. Various embodiments are therefore directed to assigning the larger number of user input interfaces (e.g., menu entries) to a plurality of objects which each individual may have fewer number of stable equilibrium states than the number of user input interfaces, but collectively provide the same or more stable equilibrium states than the number of user input interfaces.

[0119] To manage the underfitting scenario, the operation to detect 700 at least one physical object imaged in the video from the image sensor to be analyzed for use by the user to provide input toAttorney Docket P112001WO01 / 9900-112001WO01user input interfaces controlling display of content through the display, comprises to determine a required number of the user input interfaces that are to be assigned, and to select at least two physical objects for manipulation by the user to provide input to the user input interfaces, based on determining that a combination of the number of stable mechanical equilibrium states of the at least two physical objects is at least as great as the required number of the user input interfaces.

[0120] In a further operational embodiment, the operations are configured to select a set of more than one physical object for assignment of their collective stable mechanical equilibrium states, only when no single physical object in the video from the image sensor 103 is determined to have a number of stable mechanical equilibrium states that is at least as great as the required number of the user input interfaces of the application. In the context of Figure 7, the operation to select 700 at least two physical objects for manipulation by the user to provide input to the user input interfaces (based on determining that the combination of the number of stable mechanical equilibrium states of the at least two physical objects is at least as great as the required number of the user input interfaces), includes to determine that no physical object in the video has a number of stable mechanical equilibrium states that is at least as great as the required number of the user input interfaces. Thus, when no one physical object has a sufficient number of stable mechanical equilibrium states, the operation can select more than one physical object (pursuant to one or more selection rules) for assignment 730 of their collective stable mechanical equilibrium states to the required user input interfaces in order to operationally allow the user to manipulate 740 those physical objects to provide input to the user input interfaces.

[0121] In a further operational embodiment, the operations are configured to use more than one physical object based on receiving a user selection of a plurality of physical objects in the video for assignment to the user input interfaces of the application. In the context of Figure 7, the operation to select 700 at least two physical objects for manipulation by the user to provide input to the user input interfaces (based on determining that the combination of the number of stable mechanical equilibrium states of the at least two physical objects is at least as great as the required number of the user input interfaces), includes to determine receiving an indication of the user's selection of the at least two physical objects to be used for assignment to the user input interfaces. Thus, for example, the user may select (e.g., by point toward or physically touching within view of the imaging sensor 103) more than two physical objects viewed in the display 105 of the XR device 100. The operations can respond to those selections by selecting 700 those user-selected physical objects for assignment 730 of user inputAttorney Docket P112001WO01 / 9900-112001WO01interfaces to operationally allow the user to manipulate 740 those physical objects to provide input to the user input interfaces.

[0122] Determining preferred user interaction:

[0123] The operations may track a user's interaction with a physical object when providing input to assigned user input interfaces, and determine therefrom over-time a preferred way the user wants to interact with the stable mechanical equilibrium states of that particular physical object or more generally with any similarly shaped physical object.

[0124] For example, a user being left-handed may prefer to manipulate a physical object along a specific "left-to-right"-type of path as that holds certain biomechanical advantages.

[0125] The operations may determine that the user in respect to e.g., a physical object having two stable mechanical equilibriums states always, often, or rarely interacts from a first-to-second state in a clockwise manner, but in contrast respectively rarely, often, or always executes the opposite pattern direction.

[0126] Then the operations can deduct a preferred toggling / flip direction to be re-used for other more complex control physical object accessible state assignments to user input interfaces for input by the user.Further Definitions and Embodiments:

[0127] In the above-description of various embodiments of present inventive concepts, it is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of present inventive concepts. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which present inventive concepts belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense expressly so defined herein.

[0128] When an element is referred to as being "connected", "coupled", "responsive", or variants thereof to another element, it can be directly connected, coupled, or responsive to the other element or intervening elements may be present. In contrast, when an element is referred to as being "directly connected", "directly coupled", "directly responsive", or variants thereof to another element, there are no intervening elements present. Like numbers refer to like elements throughout. Furthermore,Attorney Docket P112001WO01 / 9900-112001WO01"coupled", "connected", "responsive", or variants thereof as used herein may include wirelessly coupled, connected, or responsive. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Well-known functions or constructions may not be described in detail for brevity and / or clarity. The term "and / or" includes any and all combinations of one or more of the associated listed items.

[0129] It will be understood that although the terms first, second, third, etc. may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another element / operation. Thus, a first element / operation in some embodiments could be termed a second element / operation in other embodiments without departing from the teachings of present inventive concepts. The same reference numerals or the same reference designators denote the same or similar elements throughout the specification.

[0130] As used herein, the terms "comprise", "comprising", "comprises", "include", "including", "includes", "have", "has", "having", or variants thereof are open-ended, and include one or more stated features, integers, elements, steps, components or functions but does not preclude the presence or addition of one or more other features, integers, elements, steps, components, functions or groups thereof. Furthermore, as used herein, the common abbreviation "e.g.", which derives from the Latin phrase "exempli gratia," may be used to introduce or specify a general example or examples of a previously mentioned item, and is not intended to be limiting of such item. The common abbreviation "i.e.", which derives from the Latin phrase "id est," may be used to specify a particular item from a more general recitation.

[0131] Example embodiments are described herein with reference to block diagrams and / or flowchart illustrations of computer-implemented methods, apparatus (systems and / or devices) and / or computer program products. It is understood that a block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by computer program instructions that are performed by one or more computer circuits. These computer program instructions may be provided to a processor circuit of a general purpose computer circuit, special purpose computer circuit, and / or other programmable data processing circuit to produce a machine, such that the instructions, which execute via the processor of the computer and / or other programmable data processing apparatus, transform and control transistors, values stored in memory locations, and other hardware components within such circuitry to implement theAttorney Docket P112001WO01 / 9900-112001WO01functions / acts specified in the block diagrams and / or flowchart block or blocks, and thereby create means (functionality) and / or structure for implementing the functions / acts specified in the block diagrams and / or flowchart block(s).

[0132] These computer program instructions may also be stored in a tangible computer-readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instructions which implement the functions / acts specified in the block diagrams and / or flowchart block or blocks. Accordingly, embodiments of present inventive concepts may be embodied in hardware and / or in software (including firmware, resident software, micro-code, etc.) that runs on a processor such as a digital signal processor, which may collectively be referred to as "circuitry," "a module" or variants thereof.

[0133] It should also be noted that in some alternate implementations, the functions / acts noted in the blocks may occur out of the order noted in the flowcharts. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality / acts involved. Moreover, the functionality of a given block of the flowcharts and / or block diagrams may be separated into multiple blocks and / or the functionality of two or more blocks of the flowcharts and / or block diagrams may be at least partially integrated. Finally, other blocks may be added / inserted between the blocks that are illustrated, and / or blocks / operations may be omitted without departing from the scope of inventive concepts. Moreover, although some of the diagrams include arrows on communication paths to show a primary direction of communication, it is to be understood that communication may occur in the opposite direction to the depicted arrows.

[0134] Many variations and modifications can be made to the embodiments without substantially departing from the principles of the present inventive concepts. All such variations and modifications are intended to be included herein within the scope of present inventive concepts. Accordingly, the above disclosed subject matter is to be considered illustrative, and not restrictive, and the appended examples of embodiments are intended to cover all such modifications, enhancements, and other embodiments, which fall within the spirit and scope of present inventive concepts. Thus, to the maximum extent allowed by law, the scope of present inventive concepts is to be determined by the broadest permissible interpretation of the present disclosure including the following examples ofAttorney Docket P112001WO01 / 9900-112001WO01embodiments and their equivalents, and shall not be restricted or limited by the foregoing detailed description.

Claims

Attorney Docket P112001WO01 / 9900-112001WO01CLAIMS:

1. An Extended Reality, XR, device (100) comprising a display, an image sensor (103), and a controller (101), wherein the image sensor (103) is configured to generate video imaging a field of view, FOV, of the image sensor, and wherein the controller (101) is operative to:detect at least one physical object imaged in the video from the image sensor to be analyzed for use by a user to provide input to user input interfaces controlling display of content through the display;identify a number of stable mechanical equilibrium states of the at least one physical object; responsive to determining the number of stable mechanical equilibrium states is different than a number of the user input interfaces that are to be assigned, determine assignments of individual ones of the user input interfaces to at least one of the stable mechanical equilibrium states of the at least one physical object based on a characteristic of the user input interface and / or a characteristic of the at least one of the stable mechanical equilibrium states;track, in the video from the image sensor, user manipulation of a manipulated one of the at least one physical object relative to one of the at least one of the stable mechanical equilibrium states of the manipulated one of the least one physical object; andprovide input to the user input interface assigned to the one of the at least one of the stable mechanical equilibrium states of the manipulated one of the at least one physical object, based on the tracked user manipulation.

2. The XR device (100) according to claim 1, wherein operation of the controller (101) to identify the number of stable mechanical equilibrium states of the at least one physical object, comprises, for each of the at least one physical object, to:determine an encompassing shape for the physical object, wherein the encompassing shape comprises a plurality of geometrical shapes; anddetermine the number of stable mechanical equilibrium states of the physical object based on how many of the plurality of geometrical shapes have features which are determined to be capable of supporting the physical object in a stable mechanical equilibrium state while resting on a physical surface.Attorney Docket P112001WO01 / 9900-112001WO013. The XR device (100) according to claim 2, wherein the controller (101) is further configured to determine the number of stable mechanical equilibrium states of the physical object based on which of the plurality of geometrical shapes have features matching features of stored geometrical shapes defined as capable of supporting a physical object in a stable mechanical equilibrium state while resting on a physical surface.

4. The XR device (100) according to any preceding claim, wherein operation of the controller (101) to detect at least one physical object imaged in the video from the image sensor to be analyzed for use by the user to provide input to user input interfaces controlling display of content through the display, comprises to:determine a required number of the user input interfaces that are to be assigned; and select one physical object for manipulation by the user to provide input to the user input interfaces, based on determining the number of stable mechanical equilibrium states of the one physical object is at least as great as the required number of the user input interfaces.

5. The XR device (100) according to claim 4, wherein operation of the controller (101) further comprises to:responsive to determining the number of stable mechanical equilibrium states is greater than the required number of the user input interfaces that are to be assigned, assign one of the user input interfaces to more than one of the stable mechanical equilibrium states.

6. The XR device (100) according to claim 5, wherein operation of the controller (101) to assign one of the user input interfaces to more than one of the stable mechanical equilibrium states, comprises to:identify a set of at least two stable mechanical equilibrium states of the one physical object that satisfy a rule for being spatially adjacent; andassign the one of the user input interfaces to the identified set of at least two stable mechanical equilibrium states.Attorney Docket P112001WO01 / 9900-112001WO017. The XR device (100) according to any of claims 5 to 6, wherein operation of the controller (101) to assign one of the user input interfaces to more than one of the stable mechanical equilibrium states, comprises to:identify a set of at least two stable mechanical equilibrium states of the one physical object that satisfy a rule for an amount of rotation and / or movement distance required to transition between by user manipulation of the one physical object; andassign the one of the user input interfaces to the identified set of at least two stable mechanical equilibrium states.

8. The XR device (100) according to claim 7, wherein the rule for the amount of rotation and / or movement distance required to transition between the at least two stable mechanical equilibrium states in the set by user manipulation of the one physical object, operates to perform a define one of maximizing or minimizing the amount of rotation and / or movement distance required to transition between the at least two stable mechanical equilibrium states in the set by user manipulation of the one physical object.

9. The XR device (100) according to any of claims 4 to 8, wherein operation of the controller (101) further comprises to:responsive to determining the number of stable mechanical equilibrium states is greater than the required number of the user input interfaces that are to be assigned,performing the assigning of the user input interfaces to the stable mechanical equilibrium states while reserving at least one of the stable mechanical equilibrium states that is not assigned to any of the user input interfaces, wherein the reserved at least one of the stable mechanical equilibrium states is selected to be positionally located on the one physical object between a spaced apart pair of two other stable mechanical equilibrium states that are mapped to two different user input interfaces.

10. The XR device (100) according to any of claims 4 to 9, wherein operation of the controller (101) further comprises, responsive to determining the number of stable mechanical equilibrium states is greater than the required number of the user input interfaces that are to be assigned, to:Attorney Docket P112001WO01 / 9900-112001WO01determine assignments of the user input interfaces to the stable mechanical equilibrium states of the one physical object based on comparison of probabilities that respective ones of the user input interfaces will be selected next by the user to receive user input while the XR device is operating in a present operational state.

11. The XR device (100) according to claim 10, wherein operation of the controller (101) further comprises, for each of the user input interfaces, to compute the probability based on values indicating likelihood of the XR device transitioning from the present operational state to another allowed operational state by user input provided to the respective user input interface.

12. The XR device (100) according to any of claims 10 to 11, wherein operation of the controller (101) to determine assignments of the user input interfaces to the stable mechanical equilibrium states of the one physical object based on comparison of probabilities that respective ones of the user input interfaces will be selected next by the user to receive user input while the XR device is operating in a present operational state, comprises to:assign one of the user input interfaces having the greatest probability to a greater number of the stable mechanical equilibrium states than another one of the user input interfaces have a lowest probability.

13. The XR device (100) according to any of claims 4 to 12, wherein operation of the controller (101) further comprises, responsive to determining the number of stable mechanical equilibrium states is greater than the required number of the user input interfaces that are to be assigned, to:determine assignments of the user input interfaces to the stable mechanical equilibrium states of the one physical object based on comparison of sizes of physical object surfaces, which correspond to the determined mechanical equilibrium states, to the sizes of user-selectable items that will be displayed to the user to represent the corresponding user input interfaces.

14. The XR device (100) according to any preceding claim, wherein operation of the controller (101) to detect 700 at least one physical object imaged in the video from the image sensor to be analyzed for use by the user to provide input to user input interfaces controlling display of content through the display, comprises to:Attorney Docket P112001WO01 / 9900-112001WO01determine a required number of the user input interfaces that are to be assigned; and select at least two physical objects for manipulation by the user to provide input to the user input interfaces, based on determining that a combination of the number of stable mechanical equilibrium states of the at least two physical objects is at least as great as the required number of the user input interfaces.

15. The XR device (100) according to claim 14, wherein operation of the controller (101) to select at least two physical objects for manipulation by the user to provide input to the user input interfaces, based on determining that the combination of the number of stable mechanical equilibrium states of the at least two physical objects is at least as great as the required number of the user input interfaces, is further selected based on:determining that no physical object in the video has a number of stable mechanical equilibrium states that is at least as great as the required number of the user input interfaces.

16. The XR device (100) according to any of claims 14 to 15, wherein operation of the controller (101) to select at least two physical objects for manipulation by the user to provide input to the user input interfaces, based on determining that the combination of the number of stable mechanical equilibrium states of the at least two physical objects is at least as great as the required number of the user input interfaces, is further selected based on:receiving an indication of the user's selection of the at least two physical objects to be used for assignment to the user input interfaces.

17. The XR device (100) according to any preceding claim, wherein the XR device (100) comprises a smartphone.

18. The XR device (100) according to any preceding claim, wherein the XR device (100) comprises an apparatus adapted to be worn on a user's head.

19. The XR device (100) according to any preceding claim, wherein operation of the controller (101) to provide input to the user input interface comprises providing input to an operating system executed by the XR device and / or providing input to an application executed by the XR device.Attorney Docket P112001WO01 / 9900-112001WO0120. A method by an Extended Reality, XR, device comprising a display, an image sensor, and a controller, wherein the image sensor is configured to generate video imaging a field of view, FOV, of the image sensor, the method comprising:detecting (700) at least one physical object imaged in the video from the image sensor to be analyzed for use by a user to provide input to user input interfaces controlling display of content through the display;identifying (710) a number of stable mechanical equilibrium states of the at least one physical object;responsive to determining (720) the number of stable mechanical equilibrium states is different than a number of the user input interfaces that are to be assigned, determining (730) assignments of individual ones of the user input interfaces to at least one of the stable mechanical equilibrium states of the at least one physical object based on a characteristic of the user input interface and / or a characteristic of the at least one of the stable mechanical equilibrium states;tracking (740), in the video from the image sensor, user manipulation of a manipulated one of the at least one physical object relative to one of the at least one of the stable mechanical equilibrium states of the manipulated one of the least one physical object; andproviding (750) input to the user input interface assigned to the one of the at least one of the stable mechanical equilibrium states of the manipulated one of the at least one physical object, based on the tracked user manipulation.

21. A computer program product for an Extended Reality, XR, device comprising a display, an image sensor, and a controller, wherein the image sensor is configured to generate video imaging a field of view, FOV, of the image sensor, the computer program product comprising:a computer readable medium storing instructions executable by the controller of the XR device to perform operations to:detect at least one physical object imaged in the video from the image sensor to be analyzed for use by a user to provide input to user input interfaces controlling display of content through the display;identify a number of stable mechanical equilibrium states of the at least one physical object;Attorney Docket P112001WO01 / 9900-112001WO01responsive to determining the number of stable mechanical equilibrium states is different than a number of the user input interfaces that are to be assigned, determine assignments of individual ones of the user input interfaces to at least one of the stable mechanical equilibrium states of the at least one physical object based on a characteristic of the user input interface and / or a characteristic of the at least one of the stable mechanical equilibrium states;track, in the video from the image sensor, user manipulation of a manipulated one of the at least one physical object relative to one of the at least one of the stable mechanical equilibrium states of the manipulated one of the least one physical object; andprovide input to the user input interface assigned to the one of the at least one of the stable mechanical equilibrium states of the manipulated one of the at least one physical object, based on the tracked user manipulation.

Images