System and method for identifying actions from phalange gestures

Abstract

A computer implemented method, comprising: while a user is interacting with a user interface of a computing device; receiving, from a sensor, at least one capture of at least one hand; identifying from the at least one capture, a contact of a side of a finger with a trackable or visible object, or a contact of a portion of the finger with the trackable or visible object, or a contact of a side of the portion of the finger with the trackable or visible object, or a contact of a tip of the portion of the finger with the trackable or visible object, or a contact of a hand-wear with the trackable or visible object, or a contact of a portion of the hand-wear with the trackable or visible object, or a contact of a side of the portion of the hand-wear with the trackable or visible object, or a contact of a tip of the portion of the hand-wear with the trackable or visible object; the hand-wear being attached to at least one hand; classifying a gesture in the at least one capture in accordance with the side of the finger, or with the portion of the finger, or with the side of the portion of the finger, or with the tip of the portion of the finger, or with the hand-wear, or with the portion of the hand-wear, or with the side of the portion of the hand-wear, or with the tip of the portion of the hand-wear; associating an action identification with the gesture; the association being for executing a function, corresponding to the action identification.

Description

SYSTEM AND METHOD FOR IDENTIFYING ACTIONS FROM PHALANGE GESTURESFIELD OF THE INVENTIONThe present disclosure relates to user interface in general and to analyzing gestures in particular.BACKGROUND OF THE INVENTIONComputers rely on three main input methods for scenarios with a plurality of input options such as typing and complex video games. The first method is physical buttons, which allow private and blind input. The second method is virtual buttons, which can be selected by pointing the mouse, finger or eyes at them. This method retains the user’s privacy while not requiring a dedicated device. The third method is voice input, which can be done blindly without a dedicated device.SUMMARY OF THE INVENTIONThe term computing device or computer refers herein to a device that includes a processing unit. Examples of such devices are a personal computer, a laptop, a server, a wearable device, a tablet, a cellular device, a smart TV, a video game console and an IOT (internet of things) device.In some aspects the present invention relates to a non-transitory computer-readable medium comprising instructions which, when executed by at least one processor, causes the processor to perform the method of the present invention.The term wearable headset refers herein to a computer with at least one sensor positioned in the proximity of the user’s eyes. Examples of such sensors include an RGB camera, an infrared camera and LiDAR. Examples of such devices are smart glasses and extended (augmented, mixed or virtual) reality headsets. The wearable headset may include speakers, a single display, and / or a separate display for each eye.The term character refers herein to a letter, a numeral, a symbol, a diacritic, a punctuation, a syllabogram, an emoji, a logogram, an ideogram, a pictogram, an empty space, a Unicode character or a combination thereof.The term action refers herein to an operation of the computing device as a result of user input. Examples of such actions are launching an application, displaying a menu, saving a file, adding a character to a document, playing a move in a video game, playing a musical note, playing a video or audio and shutting down the device.The term hand- wear refers herein to a device or to an object worn on the hand or on part of the hand that can be recognized based on data from at least one sensor. Examples include thimbles, rings, brass knuckles, and artificial fingernails.The term portion of a hand-wear refers herein to any part of a hand-wear that can be differentiated from the rest of the hand-wear. Examples include a distal half of a thimble, an outer side of an artificial nail, and a face of a polyhedral thimble.The term hand refers herein to a natural, an artificial and a virtual hand, or a hand with one or more artificial or virtual part.The term portion of a hand refers herein to a finger, a palm, a wrist or a back of a hand.The term finger refers herein to any digit of a hand, including the thumb and other natural and artificial and virtual digits.The term phalange refers herein to a phalange and the material surrounding the phalange, such as the skin or other soft tissue, fingernails and gloves. In artificial or virtual parts of fingers, the term refers to the location that matches the location of the phalange in a natural finger, and the material surrounding the location.The term joint refers herein to a joint and the material surrounding the joint, such as the skin or other soft tissue, and gloves. In artificial or virtual parts of fingers, the term refers to the location that matches the location of the joint in a natural finger, and the material surrounding the location. The term portion of the finger refers herein to a phalange or a joint; natural, artificial or virtual. The term side of a portion of a hand refers herein to the inner (palmar) side, the outer (dorsal) side, the radial side or the ulnar side. A hand-wear or another trackable or visible object may have a different number of sides.The term side of a portion of a finger refers herein to the inner (palmar) side, the outer (dorsal) side, the radial side or the ulnar side. A portion of a hand-wear or of another trackable or visible object may have a different number of sides.The term presenting refers herein to displaying or playing or outputting data in a way that can be sensed by a user.The term trackable or visible object refers herein to an object that can be tracked based on data from a sensor. Examples of such objects include a portion of a hand, hand-wear and a stylus. An object can be tracked, for example, for changes in its position, shape, size, color, or amount of blurriness between different frames.The term artificial trackable or visible object refers herein to a trackable or visible object that is not completely natural. Examples include hand- wear, a stylus and a portion of a hand or of a finger with one or more artificial or virtual part.The term digital media refers to any content or data that can be read by a computer, including video, sensor data, audio, images, three-dimensional models, animation, documents, databases, software and Al (artificial intelligent) generated content of all types.The term detectable mode refers herein to a certain appearance, of a trackable or visible object, that can be identified by a computing device. Examples include a straight finger, a bent finger, and a stylus with at least one folded part.

[0001] One technical problem dealt with by the present disclosure is how to provide blind input to a computer without compromising the computer’s portability and / or the user’s privacy. Smartphones, tablets and extended reality headsets typically display static buttons on the screen, which doesn’t allow for blind input. This makes users resort to methods that limit the device’s portability (like physical keyboards and game controllers) or the user’s privacy (like voice input). Even when using a desktop computer, a physical keyboard can take significant desk space, gather dust, and cause health issues like the carpal tunnel syndrome.One other technical problem disclosed by the present disclosure is how to perform a plurality of actions using hand gestures. Gesture systems typically treat each finger as a single unit, which limits the number of gestures that can be performed with the hand. Gestures are usually differentiated by the direction, duration or number of movements they require, but not by the portions or the anatomical sides of the fingers they involve.One other technical problem is how to recognize that a finger actually touches another finger or hand-wear, or another trackable or visible object, when the hand is captured from a limited number of angles as a result of the positioning of the capturing sensor or sensors.Known-in-the-art gesture recognition systems might identify a touch, from some capturing angles, when a finger hides part of another finger, which occurs when the former finger is between the capturing sensor and the latter finger. Such gesture recognition systems identify a touch even in cases when the fingers do not touch each other.One other technical problem is how to display a plurality of virtual buttons in a small region of a three-dimensional environment and still enable the user to accurately interact with the buttons. Extended reality headsets typically use the same gesture to interact with all of the virtual buttons on the screen, which requires the buttons to be distant to accurately identify which button the hand or the eye is pointing at when performing the gesture of interacting with a button. Distant buttons require larger movements of the hands or the eyes, which makes activities like typing slower and more tiring.One other technical problem is how to intuitively provide input to an extended reality headset without using an additional device like a physical keyboard or a game controller. Extended reality headsets typically rely on non-intuitive input methods, like pointing at virtual buttons without touching any solid object, or focusing the vision on a virtual button.One other technical problem is how to provide input to a wearable headset in a socially acceptable manner. For example, pointing at virtual buttons with the user’s fingers or eyes when they are visible to other people can be interpreted as pointing or staring at other people or objects.One other technical problem is how to enable disabled users to interact with a computing device. For example, users with vision impairment need to search for the keyboard every time they want to start using the keyboard. Furthermore, handheld devices such as smartphones or tablets typically use touchscreens instead of physical buttons.One technical solution is to analyze captures of phalange gestures wherein each of the phalange gestures represents an action. A user’s thumb, for example, can be used to blindly and intuitively touch the phalanges of other fingers, typically at more than one anatomical side, creating numerous distinct gestures. As another example, a stylus can be used to blindly and accurately touch the user’s phalanges, and the joints next to the phalanges. Utilizing human proprioception, the user can know where each portion of a finger is compared to other portions of hands, such as a thumb, and by extension objects worn on the hands or held by the hands, even without seeing. In addition, by using portions of fingers as buttons, the user can immediately receive different tactile feedback for each button. Even when one of the user’s thumbs cannot be used for gesturing or holding a stylus, fingers other than the thumb, such as the index finger, can be used to blindly touch the phalanges of other hands, or of the same hand’s thumb, or the joints next to the phalanges, utilizing different orientations of one hand relative to the other. These gestures can be used as a replacement for buttons for blind input when using a device with hand-tracking abilities. A gesture can be identified either from a single capture such as a photo or a frame of a video, or from a plurality of captures showing a movement or a break in a movement. Even in cases where it is difficult to identify which portion of a finger is in contact, such as using simple hand-tracking software or wearing certain types of gloves, it may still be possible to identify which anatomical side of the finger is in contact. In some embodiments, touching the same portion or side or tip of a portion of a finger with different objects can be classified as different gestures. For example, touching a side or a tip of a phalange with either hand’s thumb can input a lowercase letter, and touching the same side or tip of the phalange with another hand’s index finger can input a capital letter. In some embodiments, the object touching the portion of a finger may have a plurality of detectable modes. For example,touching a phalange with another hand’s finger while the finger is straight (indicative of using the inner side of the finger’s distal phalange) can input one letter, and touching the same phalange with the finger while the finger is bent (indicative of using the tip of the finger’s distal phalange) can input another letter.In some embodiments, holding a trackable or visible object in different ways while touching the same portion of a finger can be classified as different gestures. For example, touching a side or a tip of a phalange with a stylus while holding a stylus with the support of the index finger can input a lowercase letter, and touching the same side or tip of a phalange with a stylus while holding the stylus without the support of the index finger can input an uppercase letter. As another example, a stylus may have at least two visually distinct sides, and touching a phalange with the stylus can input either a letter or a numeral, depending on the side of the stylus captured by at least one sensor. The object may also have a plurality of detectable modes. For example, at least one part of the stylus may be foldable, and folding the part can switch between inputting letters and numerals while the stylus touches the portions of the finger. Switching between the modes may also be visually or audibly detectable.In some embodiments a user wears hand-wear to extend a finger’s reach, to replace missing phalanges or to add artificial parts to the hand to ease activities like typing in languages with many letters, such as the Khmer. The hand-wear may include a single portion, or a plurality of visually distinct portions. For example, a thimble may be bent in at least one region and act as multiple artificial phalanges. The hand-wear may change its appearance when in movement, detectable by an accelerometer, or when touched at a particular portion, tip or side, using sound, infrared, lights of different colors or motorized parts, to help identify a contact or a movement.One other technical solution is to detect a break in the movement of a finger’s visible edge, such as the tip of the distal phalange or the outer side of a joint, indicative of the finger making contact with another object. For example, a break in the movement of a finger other than the thumb can indicate that the thumb makes contact with the other finger’s inner side, or with the tip of its distal phalange. As another example, a break in the movement of the thumb can indicate that the thumb makes contact with the radial or outer side of another finger. A break in the movement of a visible edge may also include a situation when an edge is no longer visible from a certain angle during movement, indicative of the finger bending in order to touch another finger or object.One other technical solution is to use a trackable or visible object that can provide a signal such as a change in its appearance when in contact with another object. For example, a stylus may incorporate at one end a light bulb, that can emit infrared or visible light when the tip is in contact with anotherobject. The stylus may also create sound when in contact with another object, either relying on the movement of the tip or using a buzzer or a speaker.One exemplary embodiment of the disclosed subject matter is a computer implemented method, the method comprising: while a user is interacting with a user interface of a computing device; receiving, from a sensor, at least one capture of at least one hand; identifying from the at least one capture, a contact of a side of a finger with a trackable or visible object, or a contact of a portion of the finger with the trackable or visible object, or a contact of a side of the portion of the finger with the trackable or visible object, or a contact of a tip of the portion of the finger with the trackable or visible object, or a contact of a hand-wear with the trackable or visible object, or a contact of a portion of the hand-wear with the trackable or visible object, or a contact of a side of the portion of the hand-wear with the trackable or visible object, or a contact of a tip of the portion of the handwear with the trackable or visible object; the hand-wear being attached to at least one hand; classifying a gesture in the at least one capture in accordance with the side of the finger, or with the portion of the finger, or with the side of the portion of the finger, or with the tip of the portion of the finger, or with the hand- wear, or with the portion of the hand-wear, or with the side of the portion of the hand-wear, or with the tip of the portion of the hand-wear; associating an action identification with the gesture; the association being for executing a function, corresponding to the action identification. According to some embodiments the method is further comprising executing the function. According to some embodiments the method is further comprising classifying in accordance with the trackable or visible object, or with a manner in which the trackable or visible object is held, or with a detectable mode of the trackable or visible object.One other exemplary embodiment of the disclosed subject matter is a computer implemented method, comprising: while a user is interacting with a user interface of a computing device; receiving, from a sensor, a plurality of captures of at least one hand; analyzing the captures for detecting, in an at least one of the captures, a break in a movement of a finger of the hand, or a break in a movement of a hand-wear attached to the finger; identifying from the at least one capture, a contact of the finger with a trackable or visible object, or a contact of the hand- wear with a trackable or visible object, or a contact of a side of the finger with a trackable or visible object, or a contact of a portion of the finger with a trackable or visible object, or a contact of a portion of the hand-wear with a trackable or visible object, or a contact of a side of the portion of the finger with a trackable or visible object, or a contact of a tip of the portion of the finger with a trackable or visible object, or a contact of a side of the portion of the hand-wear with a trackable or visible object, or a contact of a tip of the portion of the hand-wear with a trackable or visible object; classifying a gesture inaccordance with the finger, or with the side of the finger, or with the portion of the finger, or with the portion of a hand-wear, or with the side of the portion of the finger, or with the tip of the portion of the finger, or with the side of the portion of the hand-wear, or the tip of the portion of the handwear; and associating an action-identification with the gesture; the association being for executing a function, corresponding to the action identification. According to some embodiments the method is further comprising executing the function. According to some embodiments the method is further comprising classifying in accordance with the trackable or visible object, or with a manner in which the trackable or visible object is held, or with a detectable mode of the trackable or visible object. One other exemplary embodiment of the disclosed subject matter is a computer implemented method, comprising: while a user is interacting with a user interface of a computing device; receiving, from a sensor, a plurality of captures of at least one hand; analyzing the captures for detecting, in an at least one of the captures, a break in a movement of a trackable or visible object; identifying from the at least one capture, a contact of the trackable or visible object with a portion of the hand, or a contact of the trackable or visible object with a hand-wear attached to the hand, or a contact of the trackable or visible object with a side of a finger, or a contact of the trackable or visible object with a portion of the finger, or a contact of the trackable or visible object with a portion of the hand-wear, or a contact of the trackable or visible object with a side of the portion of the finger, or a contact of the trackable or visible object with a tip of the portion of the finger, or a contact of the trackable or visible object with a side of the portion of the hand- wear, or a contact of the trackable or visible object with a tip of the portion of the hand-wear; classifying a gesture in accordance with the portion of a hand, or with the hand-wear, or with the side of a finger, or with the portion of a finger, or with the portion of a hand-wear, or with the side of the portion of the finger, or with the tip of the portion of the finger, or with the side of the portion of the hand-wear, or with the tip of the portion of the hand-wear; and associating an action-identification with the gesture; the association being for executing a function, corresponding to the action identification. According to some embodiments the method is further comprising executing the function. According to some embodiments the method is further comprising classifying in accordance with the trackable or visible object, or with a manner in which the trackable or visible object is held, or with a detectable mode of the trackable or visible object.One other exemplary embodiment of the disclosed subject matter is a computer implemented method, comprising: while a user is interacting with a user interface of a computing device; detecting a signal from at least one artificial trackable or visible object; receiving, from a sensor, at least one capture of at least one hand; identifying from the at least one capture, a contact of theartificial trackable or visible object with a portion of the hand, or a contact of the artificial trackable or visible object with a hand- wear attached to the hand, or a contact of the artificial trackable or visible object with a side of a finger, or a contact of the artificial trackable or visible object with a portion of the finger, or a contact of the artificial trackable or visible object with a portion of the hand-wear, or a contact of the artificial trackable or visible object with a side of the portion of the finger, or a contact of the artificial trackable or visible object with a tip of the portion of the finger, or a contact of the artificial trackable or visible object with a side of the portion of the hand-wear, or a contact of the artificial trackable or visible object with a tip of the portion of the hand-wear; classifying a gesture in accordance with the signal and the portion of a hand, or with the signal and the hand-wear, or with the signal and with the side of a finger, or with the signal and with the portion of a finger, or with the signal and with the portion of a hand-wear, or with the signal and with the side of the portion of the finger, or with the signal and with a tip of the portion of the finger, or with the signal and with a side of the portion of the hand-wear, or with the signal and with a tip of the portion of the hand-wear; and associating an action-identification with the gesture; the association being for executing a function, corresponding to the action identification. According to some embodiments the method is further comprising executing the function. According to some embodiments the method is further comprising classifying in accordance with the trackable or visible object, or with a manner in which the trackable or visible object is held, or with a detectable mode of the trackable or visible object.One other exemplary embodiment of the disclosed subject matter is a method, the method comprises: receiving a capture of at least one hand; analyzing the capture for identifying a location of a certain portion of a hand, a location of a certain side of a finger, a location of a certain portion of the finger, a location of a certain hand-wear, a location of a certain portion of a hand-wear, a location of a certain side of the certain portion of a finger, a location of a certain tip of the certain portion of a finger, a location of a certain side of the certain portion of a hand-wear, or a location of a certain tip of the certain portion of a hand- wear; retrieving, from a data repository, an action name or an action symbol associated with the certain portion of a hand, or with the certain side of a finger, or with the certain portion of the finger, or with the certain hand-wear, or with the certain portion of a hand-wear, or with the certain side of the certain portion of a hand, or with the certain side of the certain portion of the finger, or with the certain tip of the certain portion of the finger, or with the certain side of the certain portion of a hand-wear, or with the certain tip of the certain portion of a hand-wear; and presenting the action name or the action symbol on the certain portion of a hand, or on the certain portion of a finger, or on the certain hand-wear, or on the certain portion of the certainhand-wear, or on the certain side of the certain portion of a hand, or on the certain side of the portion of the finger, or on the certain tip of the certain portion of the finger, or on the certain side of the portion of a hand-wear, or on the certain tip of the certain portion of a hand- wear.One other exemplary embodiment of the disclosed subject matter is a method, the method comprising: receiving a plurality of digital media or receiving segments of the digital media; the digital media comprising a plurality of sides of a finger in contact with a trackable or visible object, or a plurality of portions of the finger in contact with the trackable or visible object, or a hand- wear in contact with the trackable or visible object, or a plurality of portions of the hand-wear in contact with the trackable or visible object, or a plurality of sides of a portion of the finger in contact with the trackable or visible object, or a side and a tip of the portion of the finger in contact with the trackable or visible object, or a plurality of sides of a portion of the hand-wear in contact with the trackable or visible object, or a side and a tip of the portion of the hand- wear in contact with the trackable or visible object; training an Al model with the media or the segments labeled with classifications of gestures, wherein each contact of a side of the finger with the trackable or visible object, or contact of a portion of the finger with the trackable or visible object, or contact of the hand-wear with the trackable or visible object, or contact of a portion of the hand-wear with the trackable or visible object, or contact of a side of the portion of the finger with the trackable or visible object, or contact of a tip of the portion of the finger with the trackable or visible object, or contact of a side of the portion of the hand-wear with the trackable or visible object, or contact of a tip of the portion of the hand-wear with the trackable or visible object, is classified as a certain gesture; and utilizing the Al model for identifying the gestures while a user is interacting with a computing device.One other exemplary embodiment of the disclosed subject matter is a method, the method comprises: receiving a capture of at least one hand; analyzing the capture for identifying at least one member selected from a group composed of: a number of hands, a number of fingers of a hand, a number of phalanges of a finger, an artificial trackable or visible object, a dimensional measurement of a finger, a dimensional measurement of a hand-wear, a dimensional measurement of a palm, and a distance between an edge of a hand-wear and the palm; determining a virtual button layout in accordance with the analysis; and presenting the virtual button layout to the user.Embodiments of the invention may be implemented as a computer process (method), a computing system, or as an article of manufacture, such as a computer program product or a non-transitory computer-readable media. The computer program product may be a computer storage media readable by a computer system and encoding a computer program of instructions for executing acomputer process on the computer and network devices. The computer program product may also be a propagated signal on a carrier readable by a computing system and encoding a computer program of instructions for executing a computer process.THE BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGSThe present disclosed subject matter will be understood and appreciated more fully from the following detailed description taken in conjunction with the drawings in which corresponding or like numerals or characters indicate corresponding or like components. Unless indicated otherwise, the drawings provide exemplary embodiments or aspects of the disclosure and do not limit the scope of the disclosure. In the drawings:Fig. 1 shows a block diagram of a first environment for identifying actions from phalange gestures, in accordance with some exemplary embodiments of the subject matter;Fig. 2 shows a block diagram of a second environment for identifying actions from phalange gestures, in accordance with some exemplary embodiments of the disclosed subject matter;Fig. 3 shows a flowchart diagram for identifying actions from phalange gestures, in accordance with some exemplary embodiments of the disclosed subject matter;Fig. 4 shows a flowchart diagram for presenting the hand of the user with the names or symbols of actions on the hand, in accordance with some exemplary embodiments of the disclosed subject matter;Fig. 5 shows a block diagram of a system for identifying actions from phalange gestures, in accordance with some exemplary embodiments of the subject matter;Fig. 6 shows a first example of presenting the names and the symbols of the actions on the hand of the user or on the screen, in accordance with some exemplary embodiments of the subject matter; Fig. 7 shows a second example of presenting the names and the symbols of the actions on the hand of the user or on the screen, in accordance with some exemplary embodiments of the subject matter; Fig. 8 shows a third example of presenting the names and the symbols of the actions on the hand of the user or on the screen, in accordance with some exemplary embodiments of the subject matter; Figs. 9A, 9B and 9C show examples of performing phalange gestures, in accordance with some exemplary embodiments of the subject matter;Fig. 10 shows a flowchart diagram of recommending a button layout, in accordance with some exemplary embodiments of the subject matter; andFig. 11 shows a block diagram of a third environment for identifying actions from phalange gestures, in accordance with some exemplary embodiments of the disclosed subject matter.DETAILED DESCRIPTIONReferences in the specification to “one embodiment,” “an embodiment,” “an example embodiment,” etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to affect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.If the performance of an operation is described herein as being “based on” one or more factors, it is to be understood that the performance of the operation may be based solely on such factor(s) or may be based on such factor(s) along with one or more additional factors. Thus, as used herein, the term “based on” should be understood to be equivalent to the term “based at least on.”Numerous exemplary embodiments are now described. Any section / subsection headings provided herein are not intended to be limiting. Embodiments are described throughout this document, and any type of embodiment may be included under any section / subsection. Furthermore, embodiments disclosed in any section / subsection may be combined with any other embodiments described in the same section / subsection and / or a different section / subsection in any manner.Referring now to Figure 1 showing a block diagram of a first environment for identifying actions from phalange gestures, in accordance with some exemplary embodiments of the subject matter. Environment 100 includes a computing device 110, a user 120, a user’s hand 121, a wearable accessory 130, and a sensor 140.The user 120 interacts with software that enables a plurality of input options. Examples of such options are characters in a search box, a word processor, or a spreadsheet; moves in a video game, and notes on a virtual musical instrument.The user 120 performs a gesture with his or her hand 121. The user 120 wears an accessory 130 with a sensor 140. Accessory 130 may be shaped as a necklace, with sensor 140 attached via cables to the processor (not shown in the figure), similarly to some spy cameras. The processor may be positioned behind the user’s neck alongside a battery and a communication module like a Bluetooth or Wi-Fi adapter. Accessory 130 may also be shaped, for example, as a hairbow or a badge. Sensor 140 can also be built into headphones or glasses, like in the Meta Ray-Ban glasses. Sensor 140 may also be attached to the computing device 110, or to any device that is positioned in a way that enables it to create captures that enable the computing device 110 to recognize the gesture.The computing device 110 receives a capture of the user’s hand 121 from the accessory 130 and identifies a gesture. The computing device 110 associates the gesture with an action. Examples of an action are presenting a letter in a certain language and performing an attack in a video game. In the figure, the computing device 110 presents the letter “V” in a search box 112. The computing device 110 may also present the user’s hand 121 on the screen and show action names or symbols on the screen before and / or while a gesture is performed. In some embodiments the actions are presented on an image of a hand. The computing device 101 is configured to track and identify the trackable and visible objects.Fig. 2 shows a block diagram of a second environment for identifying actions from phalange gestures, in accordance with some exemplary embodiments of the disclosed subject matter.Environment 200 includes a wearable headset 210 and a user 120. The wearable headset includes a central processor 211, a gesture recognition unit 212, memory 213 and a sensor 214. The headset 210 is attached to the head of the user 120. The sensor 214 provides captures of the user’s hands to the gesture recognition unit 212, which may be part of the main processor (e.g. Qualcomm Snapdragon XR) or a separate processor (e.g. Apple Rl). When the gesture recognition unit 212 detects a break in the movement of the edge of one or more trackable or visible object, it analyzes the capture to determine which gesture is performed by the user 120. The main processor 211 determines which action to perform based on software stored in memory 213.Fig. 3 shows a flowchart diagram for identifying actions from phalange gestures, in accordance with some exemplary embodiments of the disclosed subject matter.At block 305, a user starts an application that enables a plurality of input options or a feature that enables a plurality of input options. Examples of such an application are a word processor, a spreadsheet and a video game. An example of such a feature is a search box.At block 310, images of the user’s hands, fingers and hand-wear are sent from the sensor to the image recognition unit. The image recognition unit may be part of the main processor (e.g. Qualcomm Snapdragon XR) or a separate unit (e.g. Apple Rl).At block 315, the recognition unit may use an Al (artificial intelligence) model to detect a break in movement derived from negative acceleration of a finger’s visible edge or another trackable or visible object in one or more frames, indicative of the finger or the object making contact with another part of a hand or hand- wear, or another object. Movement may be measured relative to the sensor, or to another at least one trackable or visible object. In some embodiments the break is detected by using tools like MediaPipe Hands and OpenXR to track the visible edges of the fingers. In some embodiments, the break is detected by an additional Al model that tracks the level ofblurriness of the fingers in frames. The unit may compare the locations of the visible edges of fingers, or the level of blurriness of the fingers, between different frames. The recognition unit may determine whether the difference between the locations of the same edge of a finger in different frames, or the level of blurriness of a finger, is above a preset threshold for detecting intentional movement. The threshold may be different for each finger or edge of a finger. The thresholds may be the same for all users, or adjustable by the user or by the software, according to the user’s finger length or at least one assessment of the user’s typical degree of movement, performed with a low threshold for detecting movement. Such an assessment can also be used to detect movement impairment and adjust the button layout accordingly. The recognition unit may then determine between which frames the difference in location of the edge of the finger, or at which frame the finger’s level of blurriness, falls below the same preset threshold or a different threshold. The unit may also determine that an edge of a finger or another object is no longer visible. The unit may determine another movement by detecting another difference between the locations of the same or another edge of a finger in different frames, or an increase in the level of blurriness of the same or another finger and conclude by this other movement that the gesture is no longer performed. Such a conclusion is equivalent to the release of the button on a keyboard.In addition, an Al model can be trained to recognize the visible edges of hand-wear such as thimbles, or patterns drawn or carved onto the hand-wear. Such hand-wear can be used to extend a finger’s reach, replace missing phalanges or add artificial parts to the hand to ease activities like typing in languages with more letters than the Latin alphabet has. The hand-wear may also use an accelerometer, in some embodiments in combination with a gyroscope and a magnetometer, to detect movement and change its appearance accordingly, helping identify a break in the movement. The model can also be trained to recognize other trackable or visible objects, such as a stylus. Such objects can also change their appearance when in contact with a portion of the hand or hand- wear. The user may also use accessories to improve the hand’s visibility in low-light conditions. Examples include phosphorescent rings and a light-emitting thimble.From some capturing angles, a break in the movement of a trackable or visible object such as a finger indicates that the object makes contact with another object, and that one object doesn’t merely hide the other object in the capture. Alternatively, a signal such as a change in the appearance of a trackable or visible object can indicate that the object makes contact with a portion of a hand or hand-wear, and that the object doesn’t merely hide the portion of the hand or hand-wear in the capture. For example, the tip of a stylus may be physically attached to a hidden part with a different color, shape or level of reflectivity, and reveal it when the tip is pushed in. The tip may alsocomplete an electrical circuit when pushed in or tilted compared to the rest of the object, lighting an infrared or visible light bulb (resistive sensing). Alternatively, or additionally, the object may detect when the tip is touched by a conductive material like a portion of the hand (capacitive sensing). The object may change its appearance in different ways according to whether the tip is touched, pushed in or tilted, allowing for additional gestures. The object may also create sound at contact, using a spring mechanism, a buzzer or a speaker. Even from angles where contact can be accurately identified based on a single frame, recognizing which frames show a break in movement or a change in the appearance of a trackable or visible object and analyzing only them for specific gestures enables the software to work faster and / or on less powerful hardware. It also allows the computing device to consume less power.At block 320, the recognition unit may use at least one Al model to analyze the one or more frames, and to determine which gesture is performed by the user. A gesture can be recognized either from a hand skeleton constructed by tools like MediaPipe Hands and OpenXR, or directly from the data collected by at least one sensor, using an Al model trained with media showing phalange gestures. When relying on a hand skeleton, gestures can be differentiated by the distances between edges of fingers or other trackable or visible objects, or whether an edge is visible from a certain angle, at the time of a break in a movement. Gestures such as touching the radial side of a finger with the thumb can be recognized by the distance between the tip of the thumb’s distal phalange and the straight line between two visible edges of the finger. When the finger is captured from the inner side only, the same gestures can be recognized by the distance between the tip of the thumb’s distal phalange and the straight line between the tip of the finger’s distal phalange and another point, such as the radial side of the wrist. Some gestures, like touching the tip and the different sides of a distal phalange with the thumb, can also be differentiated by the manner in which the fingers are bent at the time of contact, according to the locations of the outer sides of their joints. Gestures that appear similar from certain angles can still be logically differentiated. For example, if the middle finger is bent compared to the index finger, then it is more likely that the same hand’s thumb touches the radial side of the middle finger than the inner side of the index finger. In some situations (for example, when the user accidentally hides an entire finger from the sensor while touching the finger’s radial side with the same hand’s thumb), it can be difficult to recognize the exact gesture from a single frame. In these situations, it is possible to use data from previous frames that shows to what degree the thumb was bent during the movement, for example.When gestures involving sides of phalanges are enabled, the sides of fingers and phalanges can be mapped according to the three-dimensional hand landmarks mapped by software like MediaPipeHands and OpenXR. The fingers other than the thumb cannot rotate independently of the rest of the hand. Therefore, when identifying a phalange, the three-dimensional hand landmarks can be utilized to determine the orientation of the phalange relative to at least one sensor, and identify that a certain side of the phalange is detectable by the sensor. Furthermore, the positions of different fingers relative to each other while a contact is detected, combined with the orientation of the hand, can assist in identifying the side of the phalange in which the contact occurred. For example, when the radial side of a hand is facing a sensor and a contact occurs between the hand’s thumb and a phalange of another finger, both fingers are visible when the thumb touches the inner side of the phalange. In a similar case, the thumb hides the phalange but does not exceed it when touching its radial side. In a similar case, the thumb exceeds the phalange when touching its outer side, and the finger that includes the phalange is typically bent to a higher degree. These characteristics can be extracted from the data collected by software such as MediaPipe Hands and OpenXR, or from an Al model trained using media containing gestures performed on different sides of phalanges, captured from different angles, for example. Such characteristics can be used for identifying a side of a phalange in which contact occurs.When a portion of the finger or the hand is touched by a trackable or visible object with visually distinct sides, the recognition unit may recognize the side of the trackable or visible object facing at least one sensor and classify the gesture accordingly. It may also recognize a detectable mode of the object, such as being folded in a certain way. The recognition unit may also use tools like MediaPipe Hands and OpenXR to determine that a user holds a trackable or visible object in a certain manner, or to determine a detectable mode for a finger that touches a portion of another hand, or hand-wear attached to the hand. For example, a certain distance between the tip of the index finger and the tip of at least one other finger, or the trackable or visible object, can indicate that the user holds the object without the support of the index finger. As another example, a bent finger can indicate that the finger touches another object with the tip of its distal phalange.At block 325, the main processor (e.g. Apple M5) associates the gesture with an action in accordance with the data repository and performs the action.Fig. 4 shows a flowchart diagram for presenting the hand of the user with the names or symbols of actions on the hand, in accordance with some exemplary embodiments of the disclosed subject matter.At block 405, a user starts an application that enables a plurality of input options or a feature that enables a plurality of input options. Examples of such an application are a word processor, a spreadsheet and a video game. An example of such a feature is a search box.At block 410, the user’s hand is captured by a sensor whose output, or information based on it, is configured to appear on the screen. In one embodiment, the user places his hand in front of a sensor whose output is preconfigured to appear on the screen by default. An example of such a sensor is the front cameras of the Apple Vision Pro mixed-reality headset. Another embodiment is when the computing device shows the sensor’s output or information based on its output on the screen while using an application that enables a plurality of input options or a feature that enables a plurality of input options. An example is a video game that determines the position of a character’s hands according to that of the user’s hands. The capturing may be continuous or periodic or one-time capturing. The screen may or may not be an independent display component. For example, at least one projector may be used to project images onto a headset’s lenses (like in Google Glass and Microsoft HoloLens) or directly onto the user’s hand. It may also be a part of the user’s field of view that can be changed or controlled by a computing device, using methods like neural signals or projecting an image directly into the user’s eye.At block 415, the recognition unit determines the location of each portion in the capture of the hand and / or the hand-wear and maps the portions of the fingers and the sides of the portions of hands and fingers. The recognition unit may also determine a detectable mode for a trackable or visible object, or that a trackable or visible object is held in a certain manner. The mapping of portions of fingers may be according to different angles in the finger’s contours, or creases in the skin or glove, depending on the Al model used. The mapping of phalanges can also be done by estimation according to structure of the human finger. One example is measuring the distance between a finger’s tip and the palm with tools such as MediaPipe Hands and OpenXR. Since phalange placement across the finger is similar between people, the system can show action names or symbols where each phalange supposedly is (near the edge, the base and the middle of the finger). This way, the system can work even when a user wears gloves. After at least one analysis of the length of the phalanges and / or hand-wear compared to one another, it is possible to recognize the location of the joints next to the phalanges even when the finger is straight and the user wears gloves. The sides of fingers and phalanges can be mapped according to the three-dimensional hand landmarks mapped by software like MediaPipe Hands and OpenXR. The fingers other than the thumb cannot rotate independently of the rest of the hand. Therefore, when identifying a phalange, the three-dimensional hand landmarks can be utilized to determine the orientation of the phalange relative to at least one sensor and identify that a certain side of the phalange is detectable by the sensor. When the palm is visible, the shortest line between the distal border of the palm, as recognized by the software, and the tip of a finger’s distal phalange, can be used for displaying names and symbols of actionsassociated with the inner side of the finger, or with its portions. The aspect ratio of the names or the symbols may change as the finger bends, as if they were drawn on the finger or the portions.Furthermore, the angle between two of these lines can be used to determine that the two fingers are spaced enough to display, between the fingers, the names or symbols of actions associated with the radial and / or ulnar sides of the fingers, or of the portions of the fingers. The width of fingers can be estimated for displaying names or symbols on their sides, as their combined width is similar to the width of the estimated distal border of the palm, and the ratios between the widths are similar among people. The distance between a visible edge of the thumb and the line between the tip of the distal phalange of another finger can be used to determine that part of the finger is hidden by the thumb, and should not have names or symbols displayed on it. The width of the user’s thumbs can be estimated from a capture of the user bringing both thumbs together.When at least one outer side of a joint of a finger is visible, the straight line between the outer side of a joint and another outer side of a joint of the finger, or the tip of the finger’s distal phalange, can be used as an approximation to a border of the finger. The fingers other than the thumb can bend only toward the palm, therefore the direction of bending can tell whether the ulnar or radial side is detectable by the sensor. The distance between two such lines can be used to determine that one finger or portion of a finger is hidden from the sensor by another, and should not have a name or a symbol displayed on its side facing the sensor. The names or symbols of actions associated with the different sides of the finger or of the portion can appear on the line between two visible edges of a finger, or on parallel lines, depending on the finger’s orientation. The aspect ratio of the displayed names or symbols may change, and they may rotate, depending on the orientation of the finger or the portion, as if they were drawn on the finger or on the portion. The thickness of the user’s fingers can be estimated from a capture of the user clapping both hands together.At block 420, the main processor (e.g. Apple M5) retrieves, from the data repository, action names or symbols associated with the gestures. The main processor also receives, from the recognition unit, the location of the portion, or side or tip of a portion of the finger, hand-wear or hand associated with each gesture. The portion, side or tip associated with the gesture is typically the portion, side or tip touched by a thumb or another trackable or visible object when performing the gesture. The main processor transmits the data to the graphical processing unit. This process may be different when using features like direct memory access (DMA).At block 425, the graphical processing unit displays the action names or symbols on the locations of the portions, sides or tips of the hand, fingers or hand-wear on the screen, each associated with the name or the symbol of the action. For example, if using MediaPipe Hands or OpenXR, when theinner side of the index finger appears on the screen, the names or symbols of the actions associated with the inner sides of the finger’s phalanges may appear between the tip of the finger’s distal phalange and the palm, each on the estimated location of the associated phalange. The names or symbols of the actions associated with the radial sides of the same phalanges may appear to the radial sides of the names or symbols associated with the inner sides. The names and symbols associated with the tip of the distal phalange or the outer sides of the finger’s phalanges may appear next to the tip of the distal phalange, in the direction opposite to the palm. When the outer side of the same finger appears on the screen, the names or symbols associated with the radial sides of the finger’s phalanges may appear on the inner side of the straight lines between two of the finger’s joints, or between the distal joint and the tip of the distal phalange. The names or symbols associated with the inner sides of the same phalanges may appear to the inner sides of the names or symbols associated with the radial sides. The names and symbols associated with the outer sides of the phalanges may appear on the outer sides of the straight lines. The names or symbols associated with the tip of the distal phalange may appear on the continuation of the straight line between the finger’s distal joint and the distal phalange’s tip.In mobile devices (phones, tablets, extended reality headsets, etc.), the graphical processing unit is normally built into the main processor. In laptop and desktop computers, it may be a separate unit that connects the main processor to the display.Fig. 5 shows a block diagram of a system for identifying actions from phalange gestures, in accordance with some exemplary embodiments of the subject matter.System 500 includes a module for detecting break in movement 501, a module for classifying gestures 502, a module for displaying action names and symbols on the screen 503 and data repository 504.The module for detecting break in movement 501 is configured for receiving a plurality of frames of the hand of a user while the user interacts with the computing device. Module 501 is further configured for analyzing the plurality of frames and for detecting a break in the movement of one or more fingers or other trackable or visible objects in one or more frames. Movement may be measured relative to the sensor, or to another at least one trackable or visible object. In one embodiment the break is detected by using tools like MediaPipe Hands and OpenXR to track the visible edges of the fingers. In another embodiment, the break is detected by an Al model that tracks the level of blurriness of the objects in frames. The module for detecting break in movement 501 may compare the locations of the visible edges of fingers or objects, or the level of blurriness of the fingers or objects, between different frames. The module for detecting break in movement 501 maydetermine whether the difference between the locations of the same edge of a finger or an object in different frames, or the level of blurriness of a finger or an object, is above a preset threshold for detecting intentional movement. The threshold may be different for each finger or edge of a finger. The thresholds may be the same for all users, or adjustable by the user or by the software, according to the user’s finger length or at least one assessment of the user’s typical degree of movement, performed with a low threshold for detecting movement. Such an assessment can also be used to detect movement impairment and adjust the button layout accordingly. The module for detecting break in movement 501 may then determine between which frames the difference in location of the edge of the finger or an object, or at which frame the level of blurriness of the finger or the object, falls below the same preset threshold or a different threshold. The module 501 may also determine that an edge of a finger or another object is no longer visible. The module for detecting break in movement 501 may determine another movement by detecting another difference between the locations of the same or another edge of a finger or an object in different frames, or an increase in the blurriness of the same or another finger or object and conclude by this other movement that the gesture is no longer performed. Such a conclusion is equivalent to the release of the button on a keyboard. In some embodiments, an Al model is used to predict a break in the movement of the fingers or objects. The model may be based on a graph neural network (GNN) and / or a transformer. It may rely on the level of blurriness in at least one capture to identify movement speed. In some embodiments, the module is configured for detecting a signal such as a change in the appearance of a trackable or visible object, indicative of the object making contact with a portion of the hand. One example is a stylus with an infrared or visible light bulb that emits light when the tip of the stylus is pushed in or tilted compared to the rest of the stylus, or touches a conductive material.The module for classifying gestures 502 is configured for receiving, from the module for detecting break in movement 501, the one or more frames in which the break in movement is detected. The module for classifying gestures 502 is further configured for feeding at least one Al (artificial intelligence) model with the images and for classifying phalange gestures from these images. The module is further configured for extracting from the data repository 504 the identification of the action that is associated with the classified gesture. If a gesture is predicted, the module may extract the identification of a preliminary action. Examples include typing a character before the associated gesture is performed, showing the character on the screen in a different way, and reading the character’s name out loud.The module 502 may identify a gesture from a hand skeleton constructed by tools like MediaPipe Hands and OpenXR. Gestures can be differentiated by the distances between edges offingers or other trackable or visible objects, or whether an edge is visible from a certain angle, at the time of a break in a movement. Gestures such as touching the radial side of a finger with the thumb can be recognized by the distance between the tip of the thumb’s distal phalange and the straight line between two visible edges of the finger. When the finger is captured from the inner side only, the same gestures can be recognized by the distance between the tip of the thumb’s distal phalange and the straight line between the tip of the finger’s distal phalange and another point, such as the radial side of the wrist. Some gestures, like touching the tip and the different sides of a distal phalange with the thumb, can also be differentiated by the manner in which the fingers are bent at the time of contact, according to the locations of the outer sides of their joints. Gestures that appear similar from certain angles can still be logically differentiated. For example, if the ring finger is bent compared to the middle finger, then it is more likely that the same hand’s thumb touches the radial side of the ring finger than the inner side of the middle finger. In some situations (for example, when the user accidentally hides an entire finger from the sensor while touching the finger’s radial side with the same hand’s thumb), it can be difficult to recognize the exact gesture from a single frame. In these situations, it is possible to use data from previous frames that show to what degree the thumb was bent during the movement, for example. The module 502 may also include an Al model that has been trained by a plurality of digital media representing gestures in which a thumb or another trackable or visible object touches the various phalanges, or the joints next to them, or hand-wear, or other portions of the hand or another hand. It is possible to train such a model with only segments of such media. For example, a model for predicting a contact may be trained using segments of video captured before the contact is made. Such an Al model may be trained by labeling the media with classifications of the associated gestures. It is possible to only label a portion of the media, and use the model to label the rest (semi-supervised machine learning). The media may contain images of natural, artificial, virtual or ALgenerated hands. In some embodiments the Al model is based on neural networks such as MobileNet, Inception, V GG, or ResNet. An existing neural network may be fine-tuned for the task, or a new model may be created.The module 502 may also use tools like MediaPipe Hands and OpenXR to determine that a user holds a trackable or visible object in a certain manner, or to determine a detectable mode for a finger that touches a portion of another hand, or hand-wear attached to the hand. For example, a certain distance between the tip of the index finger and the tip of at least one other finger, or the trackable or visible object, can indicate that the user holds the object without the support of the index finger. As another example, a straight finger can indicate that the finger touches another object with the inner side of its distal phalange.The module for displaying action names and symbols on the screen 503 is configured for displaying action names and symbols on the phalanges, or the joints next to them, or other portions of the hand and hand-wear, or other parts of the screen. For example, an action’s name or symbol can be displayed on the portion that is touched by a thumb or a stylus when performing the gesture associated with this action. The screen may or may not be an independent display component. For example, at least one projector may be used to project images onto a headset’s lenses (like in Google Glass and Microsoft HoloLens) or directly onto the user’s hand. It may also be a part of the user’s field of view that can be changed or controlled by a computing device, using methods like neural signals or projecting an image directly into the user’s eye.In one embodiment the displaying is performed by:• Mapping the regions in the hand that are associated with the actions. The mapping may be according to different angles in the finger’s contours, or creases in the skin or glove, depending on the Al model used. The mapping of phalanges can also be done by estimation. One example is measuring the distance between a finger’s tip and the palm with tools such as MediaPipe Hands and OpenXR. Since phalange placement across the finger is similar between people, the system can show action names or symbols where each phalange supposedly is (near the edge, the base and the middle of the finger). This way, the system can work even when a user wears gloves. After at least one analysis of the length of the phalanges and / or hand-wear compared to one another, it is possible to recognize the location of the joints next to the phalanges even when the finger is straight and the user wears gloves.• Extracting, from the data repository, the name or symbol of an action associated with the detected region.• Displaying the name or symbol on the detected region. For example, if using MediaPipe Hands or OpenXR, when the inner side of the index finger appears on the screen, the names or symbols of the actions associated with the inner sides of the finger’s phalanges may appear between the tip of the finger’s distal phalange and the palm, each on the estimated location of the associated phalange. The names or symbols of the actions associated with the radial sides of the same phalanges may appear to the radial sides of the names or symbols associated with the inner sides. The names and symbols associated with the tip of the distal phalange or the outer sides of the finger’s phalanges may appear next to the tip of the distal phalange, in the direction opposite to the palm. When the outer side of the same finger appears on the screen, the names or symbols associated with the radial sides of the finger’s phalanges may appear on the inner side of the straightlines between two of the finger’s joints, or between the distal joint and the tip of the distal phalange. The names or symbols associated with the inner sides of the same phalanges may appear to the inner sides of the names or symbols associated with the radial sides. The names and symbols associated with the outer sides of the phalanges may appear on the outer sides of the straight lines. The names or symbols associated with the tip of the distal phalange may appear on the continuation of the straight line between the finger’s distal joint and the distal phalange’s tip.The data repository 504 is configured for associating the classification of a gesture with an identification of the action. The data repository 504 is further configured for associating the identification of the action with a name or a symbol describing the action and with the portion of the finger, hand or hand-wear that is associated with the gesture.Fig. 6 shows a first example of presenting the names and the symbols of the actions on the hand of the user or on the screen, in accordance with some exemplary embodiments of the subject matter. In some embodiments, the system analyzes the features of the user’s hands, such as finger length or missing hands, fingers or phalanges; and recommends or chooses a suitable button layout. For example, it may be harder for a user with short thumbs to reach the different sides of the pinky finger, but easier to touch the palm with his or her thumb. Users with missing fingers or phalanges may require more buttons on each existing finger, or an alternating layout where not all buttons are shown at the same time.Figure 6 mimics the position of the fingers when using a QWERTY keyboard. The characters on the keys farthest from the user when using a keyboard are presented on the proximal phalanges. This presentation allows the fingers to be the straightest when touched, to mimic the position of the fingers when typing these characters on a keyboard.The layout is designed to minimize the movement of fingers other than the thumbs: Touching the radial side of the middle finger or the ring finger with the same hand’s thumb requires temporarily folding the finger in a way that blocks the thumb’s way to one or two other fingers, which slows down typing. For this reason, the radial sides of these fingers are used for less common actions, such as Enter, Dele te / B ackspace, and switching to an emoji keyboard or a numerals and symbols keyboard. On the pinkies, only the radial side is used, for ergonomic reasons. By folding the pinky, this side can be brought even closer to the thumb.The layout doesn’t use gestures that can be misinterpreted by other people when used in public. For example, keeping one finger straight when the rest of the fist is squeezed can be misinterpreted as pointing or “thumbs up.”For example, touching the inner side of the left-hand index finger’s phalanges inputs R, F or the numeral 4. These characters appear on the finger when its inner side is visible, and may appear next to the finger when the finger’s inner side is not visible. Touching the same phalanges from the radial side inputs T, G or the numeral 5. These characters appear on the finger when its radial side is visible, and may appear next to the finger when the finger’s radial side is not visible. Touching the left index finger’s distal phalange’s tip inputs C, and touching the outer side of the left index finger’s distal phalange inputs V. Both characters may appear next to the finger when the finger’s inner side is visible, indicating that the finger needs to be bent to different degrees to type them. Touching the palm inputs an empty space. Pulling the left thumb away from the rest of the hand’s fingers is equivalent to holding a Shift button until other input is received, and touching the palm with other fingers at the same time can input actions like Caps Lock and Tab. The locations where the user’s fingers can touch the palm can either be estimated, or recorded from at least one assessment. Squeezing the left fist switches the keyboard to a different language and clapping both hands together or squeezing both fists simultaneously closes the keyboard.Variations to the layout may not include the numerals and may include other actions instead. In some embodiments the order of the characters on each finger is reversed. In addition, areas like the radial and outer sides of the fingers (including the thumb) can be used for additional actions, like Paste, Select All, Undo and Redo, or to enable gesture combinations with the other hand, equivalent to Alt and Ctrl / Command buttons. These may also be app-specific or feature-specific actions, such as a “.com” button when typing into a web browser’s address bar.Fig. 7 shows a second example of presenting the names and the symbols of the actions on the hand of the user or on the screen, in accordance with some exemplary embodiments of the subject matter. In some embodiments, the system analyzes the features of the user’s hands, such as finger length or missing hands, fingers or phalanges; and recommends or chooses a suitable button layout. For example, it may be harder for a user with short thumbs to reach the different sides of the pinky finger, but easier to touch the palm with his or her thumb. Users with missing fingers or phalanges may require more buttons on each existing finger, or an alternating layout where not all buttons are shown at the same time.Figure 7 mimics the position of the fingers when using a numpad. The actions on the keys farthest from the user when using a numpad are presented on the proximal phalanges for allowing the fingers to be the straightest when touched. In this example, the left hand’s pinky isn’t used because its three phalanges are easy for some people to reach with the same hand’s thumb from the radialside only (without a hand- wear) while a numpad’s left-side column has four buttons (not including the 0 button).The layout may be used on its own, or as part of a layout with other actions. The layout may also be used on either hand, or on both hands at the same time. Variations to the layout may include reversing the order of the actions on each finger.Fig. 8 shows a third example of presenting the names and the symbols of the actions on the hand of the user or on the screen, in accordance with some exemplary embodiments of the subject matter. In some embodiments, the system analyzes the features of the user’s hands, such as finger length or missing hands, fingers or phalanges; and recommends or chooses a suitable button layout. For example, it may be harder for a user with long thumbs to touch the palm with his or her thumb. Users with missing fingers or phalanges may require more buttons on each existing finger, or an alternating layout where not all buttons are shown at the same time.This layout replicates the experience of typing on a smartphone, but still allows for blind input. In this layout, the radial side of each hand’s index finger is used for word suggestions.For example, touching the inner side of the right-hand index finger’s phalanges inputs P, O or I. These characters appear on the finger when its inner side is visible, and may appear next to the finger when the finger’s inner side is not visible. Touching the right index finger’s distal phalange’s tip inputs U, and touching the outer side of the right index finger’s distal phalange inputs Y. Both characters may appear next to the finger when the finger’s inner side is visible, indicating that the finger needs to be bent to different degrees to type them. Touching the palm inputs an empty space. Holding the right thumb away from the rest of the hand’s fingers turns the user’s right hand into a virtual mouse for moving the cursor. In this situation, touching the palm with other fingers can mimic the functionality of mouse buttons (left click, right click, middle click, etc.) and the other hand can show actions for selected media (Cut, Copy, Share, Search, Translate, etc.). The locations where the user’s fingers can touch the palm can either be estimated, or recorded from at least one assessment. Squeezing the right fist activates the microphone and clapping both hands together or squeezing both fists simultaneously closes the keyboard.Variations to the layout may not include the suggestions and may include other actions instead. In addition, areas like the radial and outer sides of the fingers (including the thumbs) can be used for additional actions, like Paste, Select All, Undo and Redo, or to enable gesture combinations with the other hand, equivalent to Alt and Ctrl / Command buttons. These may also be app-specific or featurespecific actions, such as a “.com” button when typing into a web browser’s address bar.Some embodiments present a telephone keypad (e.g. E.161) on either hand, or on both hands at the same time. The keypad may be used on its own, or as part of a layout with other actions.Figs. 9A, 9B and 9C show examples of performing phalange gestures, in accordance with some exemplary embodiments of the subject matter.In Figure 9A, a user touches the inner side of his or her right hand’s middle finger’s proximal phalange.In Figure 9B, a user touches the radial side of his or her left hand’s pinky finger’s intermediate phalange.In Figure 9C, a user touches his or her left hand’s index finger’s outer side of the distal phalange. Fig. 10 shows a flowchart diagram of recommending a button layout, in accordance with some exemplary embodiments of the subject matter.At block 1005, a user starts using a device that includes sensors and that operates software for tracking hands.At block 1010, the user’s hands and hand-wear, and other trackable or visible objects, are detected by the system. The user’s hands can be detected using tools like MediaPipe Hands and OpenXR. Hand-wear and other trackable or visible objects can be recognized by an Al model trained to recognize the visible edges of the specific hand-wear and objects, or patterns drawn or carved onto them. The system may also use sensors like an infrared or RGB camera, or a microphone, to recognize an infrared or visible light pattern, or a sound, that can be created by the object. The handwear or object may also have a connectivity module like Wi-Fi or Bluetooth for non-visual recognition.At block 1015, the numbers of hands, fingers, phalanges, and other trackable or visible objects; and dimensional measurements like the length of the fingers and hand-wear compared to each other, and compared to the length and width the of user’s palm, or the distance between the tip of a hand-wear and the palm, are recorded. The detection and measurement of hands, portions of hands and portions of fingers can be done with tools like MediaPipe Hands and OpenXR. The absence of phalanges can be concluded from a significant difference in length between one finger and the corresponding finger of the other hand, or an unusual length of a finger compared to the palm or to at least one other finger. An Al model can be trained to detect and measure the length of hand-wear, and other trackable or visible objects, or it can retrieve from a data repository the length of specific hand-wear and other trackable or visible objects. In some embodiments, the absolute length of the portions of the hands, portions of fingers and hand-wear can be measured by using sensors like LiDAR, or by comparing them to at least one trackable or visible object whose dimensions are known.At block 1020, the system recommends a virtual button layout based on the features detected. For example, if the ratio between the length of the user’s thumbs and the diagonal length of user’s palm is below a certain threshold, the system may recommend a layout with a button on the palm, and few buttons on the pinky finger of each hand (e.g. only on the radial side, or not at all). If a trackable or visible object like a stylus is detected, or only one hand is detected, the system may recommend a single -hand layout. If certain fingers or phalanges, or hand-wear designed to replace them, are not detected, the system may recommend a layout with more buttons on the detected fingers and phalanges, using their outer and radial sides, for example.At block 1025, the user may accept the recommended layout. The user may also choose a new assessment, a different preset layout, or to change the layout manually.Fig. 11 shows a block diagram of a third environment for identifying actions from phalange gestures, in accordance with some exemplary embodiments of the disclosed subject matter.Environment 1100 includes a user’s hand 121, and an artificial trackable or visible object 1110. The artificial trackable or visible object 1110 may provide a signal when it touches the user’s hand 121 or hand-wear. For example, a stylus may incorporate at one end a light bulb, that can emit infrared or visible light when the tip is in contact with another object. The stylus may also create sound when in contact with another object, either relying on the movement of the tip or using a buzzer or a speaker. These signals can be detected by a computing device (not shown in the figure), using sensors like an infrared or RGB camera, or a microphone.For example, the tip of a stylus may be physically attached to a hidden part with a different color, shape or level of reflectivity, and reveal it when the tip is pushed in. The tip may also complete an electrical circuit when pushed in or tilted compared to the rest of the object 1110, lighting an infrared or visible light bulb (resistive sensing). Alternatively, or additionally, the object 1110 may detect when the tip is touched by a conductive material like a portion of the hand (capacitive sensing). The object 1110 may change its appearance in different ways according to whether the tip is touched, pushed in or tilted, allowing for additional gestures.A computing device may then use hand-tracking software, and potentially at least one other Al model, to analyze at least one capture of the hand 121 taken by a sensor like an RGB or infrared camera around the time of signal reception, and determine which portion of the hand 121, finger or hand-wear is touched by the object 1110, and in which manner the object 1110 is held. The computing device may also determine a detectable mode of object 1110. The computing device may use at least one Al model to recognize the visible edges of the trackable of visible object 1110, and the hand-wear worn by the user. The model can be trained to recognize visually distinct sides of thetrackable or visible object 1110, or detectable modes, like folding a portion of the object. Switching between the modes may also be visually or audibly detectable, so the computing device may inform the user of the switch.Tools like MediaPipe Hands and OpenXR can be used to identify the portion of the hand 121 or portion of the finger touched by the trackable or visible object. After at least one assessment of the length of the user’s fingers compared to the palm or the trackable or visible object 1110, it is possible to determine which portion of a finger is touched even when the trackable or visible object 1110, or the hand that holds it, hides the tip of the finger. The computing device may also use tools like MediaPipe Hands and OpenXR to determine that a user holds the trackable or visible object 1110 in a certain manner. For example, a certain distance between the tip of the index finger and the tip of at least one other finger, or the trackable or visible object 1110, can indicate that the user holds the object without the support of the index finger.In some embodiments, holding the trackable or visible object 1110 in different manners while touching the same portion of a finger can be classified as different gestures. For example, touching a side or a tip of a phalange with a stylus while holding a stylus with the support of the index finger can input a lowercase letter, and touching the same side or tip of a phalange with a stylus while holding the stylus without the support of the index finger can input an uppercase letter. As another example, touching a phalange with the stylus can input either a letter or a numeral, depending on the side of the stylus captured by at least one sensor. Switching between detectable modes, like folding a part of the object 1110, can also switch between inputting letters and numerals while the object touches the same portions of the finger.The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.It should be noted that, in some alternative implementations, the functions noted in the block of a figure may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.

Claims

What is claimed is:

1. A computer implemented method, comprising:while a user is interacting with a user interface of a computing device; receiving, from a sensor, at least one capture of at least one hand;identifying from said at least one capture, a contact of a side of a finger with a trackable or visible object, or a contact of a portion of said finger with said trackable or visible object, or a contact of a side of said portion of said finger with said trackable or visible object, or a contact of a tip of said portion of said finger with said trackable or visible object, or a contact of a hand- wear with said trackable or visible object, or a contact of a portion of said hand-wear with said trackable or visible object, or a contact of a side of said portion of said hand- wear with said trackable or visible object, or a contact of a tip of said portion of said hand-wear with said trackable or visible object; said hand-wear being attached to at least one hand;classifying a gesture in said at least one capture in accordance with said side of said finger, or with said portion of said finger, or with said side of said portion of said finger, or with said tip of said portion of said finger, or with said hand-wear, or with said portion of said hand-wear, or with said side of said portion of said hand-wear, or with said tip of said portion of said hand-wear.associating an action identification with said gesture;said association being for executing a function, corresponding to said action identification.

2. A computer implemented method, comprising:while a user is interacting with a user interface of a computing device; receiving, from a sensor, a plurality of captures of at least one hand;analyzing said captures for detecting, in an at least one of said captures, a break in a movement of a finger of said hand, or a break in a movement of a hand-wear attached to said finger;identifying from said at least one capture, a contact of said finger with a trackable or visible object, or a contact of said hand- wear with a trackable or visible object, or a contact of a side of said finger with a trackable or visible object, or a contact of a portion of said finger with a trackable or visible object, or a contact of a portion of said hand- wear with a trackable or visible object, or a contact of a side of said portion of said finger with a trackable or visible object, or a contact of a tip of said portion of said finger with a trackableor visible object, or a contact of a side of said portion of said hand-wear with a trackable or visible object, or a contact of a tip of said portion of said hand-wear, with a trackable or visible object;classifying a gesture in accordance with said finger, or with said side of said finger, or with said portion of said finger, or with said portion of a hand- wear, or with said side of said portion of said finger, or with said tip of said portion of said finger, or with said side of said portion of said hand- wear, or said tip of said portion of said hand-wear; and associating an action-identification with said gesture;said association being for executing a function, corresponding to said action identification.

3. A computer implemented method, comprising:while a user is interacting with a user interface of a computing device; receiving, from a sensor, a plurality of captures of at least one hand;analyzing said captures for detecting, in an at least one of said captures, a break in a movement of a trackable or visible object;identifying from said at least one capture, a contact of said trackable or visible object with a portion of said hand, or a contact of said trackable or visible object with a hand-wear attached to said hand, or a contact of said trackable or visible object with a side of a finger, or a contact of said trackable or visible object with a portion of said finger, or a contact of said trackable or visible object with a portion of said hand-wear, or a contact of said trackable or visible object with a side of said portion of said finger, or a contact of said trackable or visible object with a tip of said portion of said finger, or a contact of said trackable or visible object with a side of said portion of said hand-wear, or a contact of said trackable or visible object with a tip of said portion of said hand-wear;classifying a gesture in accordance with said portion of a hand, or with said handwear, or with said side of a finger, or with said portion of a finger, or with said portion of a hand-wear, or with said side of said portion of said finger, or with said tip of said portion of said finger, or with said side of said portion of said hand-wear, or with said tip of said portion of said hand-wear; andassociating an action-identification with said gesture;said association being for executing a function, corresponding to said action identification.

4. A computer implemented method, comprising:while a user is interacting with a user interface of a computing device;detecting a signal from at least one artificial trackable or visible object; receiving, from a sensor, at least one capture of at least one hand;identifying from said at least one capture, a contact of said artificial trackable or visible object with a portion of said hand, or a contact of said artificial trackable or visible object with a hand- wear attached to said hand, or a contact of said artificial trackable or visible object with a side of a finger, or a contact of said artificial trackable or visible object with a portion of said finger, or a contact of said artificial trackable or visible object with a portion of said hand- wear, or a contact of said artificial trackable or visible object with a side of said portion of said finger, or a contact of said artificial trackable or visible object with a tip of said portion of said finger, or a contact of said artificial trackable or visible object with a side of said portion of said hand-wear, or a contact of said artificial trackable or visible object with a tip of said portion of said hand-wear;classifying a gesture in accordance with said signal and said portion of a hand, or with said signal and said hand-wear, or with said signal and with said side of a finger, or with said signal and with said portion of a finger, or with said signal and with said portion of a hand-wear, or with said signal and with said side of said portion of said finger, or with said signal and with a tip of said portion of said finger, or with said signal and with a side of said portion of said hand-wear, or with said signal and with a tip of said portion of said handwear; andassociating an action-identification with said gesture;said association being for executing a function, corresponding to said action identification.

5. The method of claim 1, further comprising executing said function.

6. The method of claim 2, further comprising executing said function.

7. The method of claim 3, further comprising executing said function.

8. The method of claim 4, further comprising executing said function.

9. The method of claim 1, further comprising classifying in accordance with said trackable or visible object, or with a manner in which said trackable or visible object is held, or with a detectable mode of said trackable or visible object.

10. The method of claim 2, further comprising classifying in accordance with said trackable or visible object, or with a manner in which said trackable or visible object is held, or with a detectable mode of said trackable or visible object.

11. The method of claim 3, further comprising classifying in accordance with said trackable or visible object, or with a manner in which said trackable or visible object is held, or with a detectable mode of said trackable or visible object.

12. The method of claim 4, further comprising classifying in accordance with said trackable or visible object, or with a manner in which said trackable or visible object is held, or with a detectable mode of said trackable or visible object.

13. A method, the method comprises:receiving a capture of at least one hand;analyzing said capture for identifying a location of a certain portion of a hand, or a location of a certain side of a finger, or a location of a certain portion of said finger, or a location of a certain hand-wear, or a location of a certain portion of a hand-wear, or a location of a certain side of said certain portion of a finger, or a location of a certain tip of said certain portion of a finger, or a location of a certain side of said certain portion of a hand-wear, or a location of a certain tip of said certain portion of a hand- wear;retrieving, from a data repository, an action name or an action symbol associated with said certain portion of a hand, or with said certain side of a finger, or with said certain portion of said finger, or with said certain hand- wear, or with said certain portion of a handwear, or with said certain side of said certain portion of a finger, or with said certain tip of said certain portion of a finger, or with said certain side of said certain portion of a handwear, or with said certain tip of said certain portion of a hand-wear; andpresenting said action name or said action symbol on said certain portion of a hand, or on said certain portion of a finger, or on said certain hand-wear, or on said certain portion of said certain hand- wear, or on said certain side of said certain portion of a hand, or on said certain side of said portion of said finger, or on said certain tip of said certain portion of said finger, or on said certain side of said portion of a hand-wear, or on said certain tip of said certain portion of a hand- wear.

14. A method, the method comprising:receiving a plurality of digital media or receiving segments of said digital media; said digital media comprising a plurality of sides of a finger in contact with a trackable or visible object, or a plurality of portions of said finger in contact with said trackable or visible object, or a hand- wear in contact with said trackable or visible object, or a plurality of portions of said hand- wear in contact with said trackable or visible object, or a plurality of sides of a portion of said finger in contact with said trackable or visible object, or a side and a tip of said portion of said finger in contact with said trackable or visible object, or a plurality of sides of a portion of said hand-wear in contact with said trackable or visible object, or a side and a tip of said portion of said hand-wear in contact with said trackable or visible object; training an Al model with said media or said segments labeled with classifications of gestures, wherein each said contact of a side of said finger with said trackable or visible object, or contact of a portion of said finger with said trackable or visible object, or contact of said hand-wear with said trackable or visible object, or contact of a portion of said handwear with said trackable or visible object, or contact of a side of a portion of said finger with said trackable or visible object, or contact of a tip of said portion of said finger with said trackable or visible object, or contact of a side of said portion of said hand-wear with said trackable or visible object, or contact of a tip of said portion of said hand-wear with said trackable or visible object, is classified as a certain gesture; andutilizing said Al model for identifying said gestures while a user is interacting with a computing device.

15. A method, the method comprises:receiving a capture of at least one hand;analyzing said capture for identifying at least one member selected from a group composed of: a number of hands, a number of fingers of a hand, a number of phalanges of a finger, an artificial trackable or visible object, a dimensional measurement of a finger, a dimensional measurement of a hand-wear, a dimensional measurement of a palm, and a distance between an edge of a hand- wear and said palm;determining a virtual button layout in accordance with said analysis; and presenting said virtual button layout to said user.

Images