Systems and methods for user interfaces with pointing angle and user-intention estimation

Abstract

Systems and methods for estimating an orientation of a conductive object and / or an estimated user-intended pointing location for interacting with an apparatus operable with a touchless interface. A plurality of time-sequenced single reference center of mass points is determined based on one or more capacitive sensor signals. Each single reference center of mass point corresponds to a location of a center of mass of the conductive object and is associated with a time stamp. One or more machine learning inputs are generated based on the plurality of time-sequenced single reference center of mass points. A trained machine learning algorithm, when executed with the one or more machine learning inputs, outputs an estimated zenith angle and an estimated azimuthal angle of the conductive object. The estimated user- intended pointing location is based at least on the estimated zenith angle and estimated azimuthal angle of the conductive object.

Description

SYSTEMS AND METHODS FOR USER INTERFACES WITH POINTING ANGLE AND USER-INTENTION ESTIMATIONCross-Reference to Related Applications

[0001] This application claims priority from, and for the purposes of the United States of America the benefit under 35 USC 119 in connection with, United States application No. 63 / 761 ,874 filed 21 February 2025 which is hereby incorporated herein by reference.Field

[0002] This technology relates to touchless or hybrid touch / touchless interaction with machines and the like. Particular embodiments provide systems and methods for estimating an orientation of a conductive object and / or estimating a user-intended pointing location for controlling an apparatus operable by a touchless interface.Background

[0003] The world is saturated with touch-based interfaces - including touchscreens, keypads, light switches, elevator buttons, etc. These interfaces are inexpensive to produce and generally intuitive to use. Other types of interactive interface include touchless interfaces and hybrid touch / touchless interfaces. However, human-machine interfaces (HMI) of the touchless, or hybrid touch / touchless, variety are currently limited primarily to industry-specific niche applications, such as for gaming, entertainment, automobile panels, home automation, and clinical sterile interactions.

[0004] One type of touchless solution for HMI systems is far-range gesture-based interfaces. Far-range gesture-based interfaces typically have operational range in the range of 1-6 meters and utilize 3D sensors to generate 3D point clouds. However, far-range gesture-based interfaces have long been limited by a number of key usability issues: learning curve and fatigue. Typically, users need to memorize a set of gestures to control an apparatus through the far-range gesture-based interfaces. Some of the gestures may be unintuitive to users and therefore impose a steep learning curve. In addition, the larger physiological movements often required for interaction with far-range gesture-based interfaces can lead to physical fatigue (e.g. arm fatigue, etc.) after prolonged use. Lastly, precise interaction with HMI elements by a user may be challenging when the user is positioned at a distance from the far-range gesture-based interface. For example, users may experience difficulty in aimingfor small targets on a display via far-range gesture-based interfaces.

[0005] Another type of touchless solution for HMI systems is near-apparatus touchless interface, where the detection range is relatively close to the apparatus (e.g. within 30 centimeters of the near-apparatus touchless interface). Near-apparatus touchless interaction ameliorates some of the issues associated with far-range gesture-based interfaces by allowing users to interact at a much more proximate position relative to the touchless interface. The proximity of the user’s body parts to the touchless interface makes precision targeting more intuitive and accurate compared to far-range gestured-based interfaces. Moreover, near-apparatus touchless interfaces are also able to facilitate hybrid touch / touchless interfaces -allowing for a combination of touch and touchless input to interact with the interface. There are a variety of applications for near-apparatus touchless interfaces in a variety of industries: medical (sterile surgical environments, bedside monitors); industrial (factories, clean rooms, automation control); outdoor environment (kiosks, ATMs, transit systems, outdoor ticketing); military (protected control panels, secure command systems, rugged tablets); retail (self-checkout kiosks, interactive displays), etc.

[0006] However, near-apparatus touchless interfaces also have their own challenges. In near-apparatus touchless interfaces, the presence of additional conductive mass (e.g. fingers other than the pointing finger, fist / palm, etc.) other than the conductive object intended for interaction (e.g. the tip of the pointing finger) with the touchless interface may affect the accuracy in estimating user intention in relation to the apparatus operable by a touchless interface. For example, the presence of the additional conductive mass may cause distortions in sensor signals (e.g. capacitive sensor signals) that lead to discrepancies between an estimated used intended target location and the actual user intended target location.

[0007] Many near-apparatus touchless interfacing technologies utilize capacitive sensing to detect presence of conductive objects (e.g. human body parts). For example, capacitive sensors that utilize electrodes could be placed on transparent conductive film (which may be overlaid on top of a screen of an interface apparatus) to detect a center of mass of a conductive object (typically a user’s hand and finger) as a single reference point within a detection range of the capacitive sensors.

[0008] Since capacitive-based near-apparatus touchless interfaces typically have a detection range that is relatively proximate (e.g. 30 centimeters or less) to aninteractive surface of an apparatus operable by touchless interface, users typically interact with near-range touchless interfaces in manners similar to how they interact with touch-based interactive surfaces. For example, users may use their hand and especially their finger to execute motions (e.g. pointing, swiping, twirling, etc.) to indicate user intention.

[0009] However, while the user intention is typically closely associated with a specific conductive object (e.g. tip of a pointing finger), additional conductive mass (e.g. remaining portions of the pointing finger as well as other portions of the hand such as other fingers, fist / palm, etc.) are typically also within the detection range of the capacitive sensors in near-apparatus touchless interface applications. As a result, the single reference center of mass point as detected by the capacitive sensors may be distorted by the presence of such additional conductive mass and consequently could affect accurate determination of user intention. The effect of the additional conductive mass becomes generally more pronounced when the additional conductive mass is positioned more proximate to the capacitive sensors.

[0010] Hence, there is a desire for improved touchless near-apparatus system that provides intuitive user experiences and facilitates accurate apparatus response. Specifically, there is a desire to provide touchless interactions that can account for misalignment issues due to presence of additional conductive masses within a detection range of the touchless interface.Summary

[0011] One aspect of the invention provides a method for determining an orientation of a conductive object for interaction with an apparatus operable with a touchless interface. The method comprises: receiving one or more capacitive sensor signals from one or more capacitive sensors, the one or more capacitive sensors sensitive to disturbances in their electric fields caused by a conductive object in proximity thereto; determining, based on the one or more capacitive sensor signals, a plurality of time-sequenced single reference center of mass points, each single reference center of mass point corresponding to a location of a center of mass of the conductive object as a single reference point detected by the one or more capacitive sensors and associated with a time stamp; generating one or more machine learning inputs based on the plurality of time-sequenced single reference center of mass points; and, providing the one or more machine learning inputs to a trained machine learning algorithm comprising trainable parameters, that when executed, outputs an estimatedzenith angle and an estimated azimuthal angle of the conductive object. The estimated zenith angle is measured from a zenith reference axis extending perpendicularly to a user-facing surface of the apparatus and the estimated azimuthal angle is measured from an azimuthal reference axis extending in parallel to the userfacing surface of the apparatus.

[0012] The one or more machine learning inputs may comprise at least one of: at least a portion of the plurality of time-sequenced single reference center of mass points; an estimated trajectory of the conductive object; average velocity value(s) of the conductive object; and, estimated acceleration value(s) of the conductive object.

[0013] The method may comprise determining the estimated trajectory based on the plurality of time-sequenced single reference center of mass points.

[0014] Determining the estimated trajectory may comprise, at least in part, tracing the plurality of time-sequenced single reference center of mass points in chronological order.

[0015] The method may comprise calculating the average velocity value between a pair of time-sequenced single reference center of mass points based on a distance between the pair of single reference center of mass points and a difference in the associated time stamps.

[0016] Each average velocity value may be associated with a corresponding estimated velocity time stamp.

[0017] The estimated velocity time stamp may be determined as the temporal midpoint between the time stamps used to calculate the average velocity value.

[0018] The estimated acceleration value between a pair of average velocity values may be calculated based on a difference between the pair of average velocity values and a difference between the associated estimated velocity time stamps.

[0019] The machine learning algorithm may be trained to identify stopping motions based at least on the estimated acceleration value(s). The stopping motions may comprise motions of the conductive object during which a velocity of the conductive object decreases to substantially zero.

[0020] The method may comprise continuously determining the plurality of time-sequenced single reference center of mass points within successive time windows. Each time window may be of a predetermined / configurable time duration, to generate the one or more machine learning inputs associated with each corresponding time window.

[0021] A starting time of each of the successive time windows may be offset from a starting time of a preceding time window by a predetermined / configurable temporal offset.

[0022] The method may comprise training the machine learning model. Training the machine learning model may comprise: initializing values for the trainable parameters; and performing a plurality of training iterations, each training iteration comprising: (i) determining, for each motion in a set of labelled motion training data, predicted azimuthal angle and predicted zenith angle based on current values for the trainable parameters of the machine learning model; (ii) calculating a loss value based on a comparison between the predicted azimuthal angle and predicted zenith angle and corresponding ground truth azimuthal angle and ground truth zenith angle from the labelled motion training data; (iii) computing gradients of the loss value with respect to the trainable parameters; and, (iv) updating the current values for the trainable parameters based on the computed gradients.

[0023] The motion training data for training the machine learning algorithm may comprise training data generated by a robotic arm with joints and an end-effector to simulate human motions.

[0024] The robotic arm may be programmed to simulate human motions of interacting with a near-apparatus touchless interface.

[0025] The robotic arm may be programmed to generate varied training data by randomizing simulated trajectories of the end-effector.

[0026] The simulated human motions may comprise one or more of: swiping, finger twirling, finger tapping, touching, finger hovering, finger dragging.

[0027] The motion training data may comprise datasets with simulated fixed-distance motions. The simulated fixed-distance motions may comprise motions of the endeffector of the robotic arm stopping at or moving relative to a user-facing side of the apparatus at fixed distances to thereby emulate fixed-distance motions of a conductive object interacting with the user-facing side of the apparatus at fixed distances.

[0028] The machine learning algorithm may be trained on datasets comprising the simulated fixed-distance motions to thereby develop the trainable parameters capable of identify fixed-distance motions of the conductive object.

[0029] The method may comprise detecting user interactions associated with the fixed-distance motions as eliciting a touch-like response from an interactive elementon the user-facing side of the apparatus.

[0030] Another aspect of the invention provides a system for estimating an orientation of a conductive object for interaction with an apparatus operable with a touchless interface. The system comprises: one or more capacitive sensors which are sensitive to disturbances in their electric fields caused by a conductive object in proximity thereto; and a processor connected to receive one or more capacitive sensor signals from the one or more capacitive sensors. The processor is configured to: determine a plurality of time-sequenced single reference center of mass points based on the one or more capacitive sensor signals, each single reference center of mass point corresponding to a location of a center of mass of the conductive object as a single reference point and a corresponding time stamp; generate one or more machine learning inputs based on the plurality of time-sequenced single reference center of mass points; and execute a trained machine learning algorithm with the one or more machine learning inputs to infer an estimated zenith angle and an estimated azimuthal angle of the conductive object. The estimated zenith angle is measured from a zenith reference axis extending perpendicularly to a user-facing surface of the apparatus and the estimated azimuthal angle is measured from an azimuthal reference axis extending in parallel to the user-facing surface of the apparatus.

[0031] The apparatus may comprise a protective layer overlaid on the capacitive sensors.

[0032] The protective layer may be spaced apart from a user-facing side of the apparatus by a separation distance to define an air gap between the protective layer and the user-facing side of the apparatus to thereby protect the user-facing side of the apparatus from physical damage due to deformation of the protective layer (e.g. when an external force is applied against the protective layer) and / or to provide heat insulation to the user-facing side of the apparatus from ambient heat.

[0033] Motion training data for the machine learning algorithm may comprise datasets with simulated motions emulating a conductive object coming into contact with the protective layer when interacting with the touchless interface to thereby train the machine learning algorithm to identify motion patterns specific to conductive object coming into contact with the protective layer.

[0034] The protective layer may comprise optically transparent material.

[0035] The protective layer may be made of fiber optic glass. The apparatus may comprise a display configured to display images wherein the displayed images areoptically transmitted by the fiber optic glass to a user-interactive surface of the protective layer.

[0036] The protective layer may comprise optically opaque material.

[0037] Outputs of the apparatus operable by the touchless interface may be external to the opaque protective layer.

[0038] The system may comprise light signal generator(s) (e.g. LEDs) wherein the outputs of the apparatus comprise light signals (e.g. LED light signals).

[0039] The processor may be configured to determine a user-intended pointing location on the user-facing surface of the apparatus based at least on the estimated zenith angle and estimated azimuthal angle of the conductive object.

[0040] The processor may be configured to generate a control signal for the apparatus based at least on the determined user-intended pointing location.

[0041] The system may comprise any of the features, combinations of features and / or sub-combinations of features described herein.

[0042] Another aspect of the invention provides a method for estimating a userintended pointing location on a user-facing surface of an apparatus operable with a touchless interface. The method comprises: receiving one or more capacitive sensor signals from one or more capacitive sensors, the one or more capacitive sensors sensitive to disturbances in their electric fields caused by a conductive object in proximity thereto; determining, based on the one or more capacitive sensor signals, a plurality of time-sequenced single reference center of mass points, each single reference center of mass point corresponding to a location of a center of mass of the conductive object as a single reference point detected by the one or more capacitive sensors and associated with a time stamp; generating one or more machine learning inputs based on the plurality of time-sequenced single reference center of mass points; providing the one or more machine learning inputs to a trained machine learning algorithm comprising trainable parameters, that when executed, outputs an estimated zenith angle and an estimated azimuthal angle of the conductive object; and, determining the user-intended pointing location on the user-facing surface of the apparatus based at least on the estimated zenith angle and the estimated azimuthal angle of the conductive object. The estimated zenith angle may be measured from a zenith reference axis extending perpendicularly to the user-facing surface of the apparatus and the estimated azimuthal angle may be measured from an azimuthal reference axis extending in parallel to the user-facing surface of the apparatus.

[0043] The one or more machine learning inputs may comprise at least one of: at least a portion of the plurality of time-sequenced single reference center of mass points; an estimated trajectory of the conductive object; average velocity value(s) of the conductive object; and, estimated acceleration value(s) of the conductive object.

[0044] The method may comprise determining the estimated trajectory based on the plurality of time-sequenced single reference center of mass points.

[0045] Determining the estimated trajectory may comprise, at least in part, tracing the plurality of time-sequenced single reference center of mass points in chronological order.

[0046] The method may comprise calculating the average velocity value between a pair of time-sequenced single reference center of mass points based on a distance between the pair of single reference center of mass points and a difference in the associated time stamps.

[0047] Each average velocity value may be associated with a corresponding estimated velocity time stamp.

[0048] The estimated velocity time stamp may be determined as the temporal midpoint between the time stamps used to calculate the average velocity value.

[0049] The estimated acceleration value between a pair of average velocity values may be calculated based on a difference between the pair of average velocity values and a difference between the associated estimated velocity time stamps.

[0050] The machine learning algorithm may be trained to identify stopping motions based at least on the estimated acceleration value(s). The stopping motions may comprise motions of the conductive object during which a velocity of the conductive object decreases to substantially zero.

[0051] The method may comprise continuously determining the plurality of time-sequenced single reference center of mass points within successive time windows, each time window of a predetermined / configurable time duration, to generate the one or more machine learning inputs associated with each corresponding time window.

[0052] A starting time of each of the successive time windows may be offset from a starting time of a preceding time window by a predetermined / configurable temporal offset.

[0053] The method may comprise training the machine learning model. Training the machine learning model may comprises initializing values for the trainable parameters; and performing a plurality of training iterations, each training iterationcomprising: (i) determining, for each motion in a set of labelled motion training data, a predicted pointing location based on current values for the trainable parameters of the machine learning model; (ii) calculating a loss value based on a comparison between the predicted pointing location and corresponding ground truth pointing location from the labelled motion training data; (iii) computing gradients of the loss value with respect to the trainable parameters; and, (iv) updating the current values for the trainable parameters based on the computed gradients.

[0054] The motion training data for training the machine learning algorithm may comprise training data generated by a robotic arm with joints and an end-effector to simulate human motions.

[0055] The robotic arm may be programmed to simulate human motions of interacting with a near-apparatus touchless interface.

[0056] The robotic arm may be programmed to generate varied training data by randomizing simulated trajectories of the end-effector.

[0057] The simulated human motions may comprise one or more of: swiping, finger twirling, finger tapping, touching, finger hovering, finger dragging.

[0058] The motion training data may comprise datasets with simulated fixed-distance motions. The simulated fixed-distance motions may comprise motions of the endeffector of the robotic arm stopping at or moving relative to a user-facing side of the apparatus at fixed distances to thereby emulate fixed-distance motions of a conductive object interacting with the user-facing side of the apparatus at fixed distances.

[0059] The machine learning algorithm may be trained on datasets comprising the simulated fixed-distance motions to thereby develop the trainable parameters capable of identify fixed-distance motions of the conductive object.

[0060] The method may comprise detecting user interactions associated with the fixed-distance motions as eliciting a touch-like response from an interactive element on the user-facing side of the apparatus.

[0061] Another aspect of the invention provides a system for estimating an orientation of a conductive object for interaction with an apparatus operable with a touchless interface. The system comprises: one or more capacitive sensors which are sensitive to disturbances in their electric fields caused by a conductive object in proximity thereto; and a processor connected to receive one or more capacitive sensor signals from the one or more capacitive sensors. The processor is configured to: determine aplurality of time-sequenced single reference center of mass points based on the one or more capacitive sensor signals, each single reference center of mass point corresponding to a location of a center of mass of the conductive object as a single reference point and a corresponding time stamp; generate one or more machine learning inputs based on the plurality of time-sequenced single reference center of mass points; execute a trained machine learning algorithm with the one or more machine learning inputs to infer an estimated zenith angle and an estimated azimuthal angle of the conductive object; and, determine the user-intended pointing location on the user-facing surface of the apparatus based at least on the estimated zenith angle and the estimated azimuthal angle of the conductive object. The estimated zenith angle may be measured from a zenith reference axis extending perpendicularly to the user-facing surface of the apparatus and the estimated azimuthal angle may be measured from an azimuthal reference axis extending in parallel to the user-facing surface of the apparatus.

[0062] The apparatus may comprise a protective layer overlaid on the capacitive sensors.

[0063] The protective layer may be spaced apart from a user-facing side of the apparatus by a separation distance to define an air gap between the protective layer and the user-facing side of the apparatus to thereby protect the user-facing side of the apparatus from physical damage due to deformation of the protective layer (e.g. when an external force is applied against the protective layer) and / or to provide heat insulation to the user-facing side of the apparatus from ambient heat.

[0064] Motion training data for the machine learning algorithm may comprise datasets with simulated motions emulating a conductive object coming into contact with the protective layer when interacting with the touchless interface to thereby train the machine learning algorithm to identify motion patterns specific to conductive object coming into contact with the protective layer.

[0065] The protective layer may comprise optically transparent material.

[0066] The protective layer may be made of fiber optic glass and the apparatus may comprise a display configured to display images wherein the displayed images are optically transmitted by the fiber optic glass to a user-interactive surface of the protective layer.

[0067] The protective layer may comprise optically opaque material.

[0068] Outputs of the apparatus operable by the touchless interface may be externalto the opaque protective layer.

[0069] The system may comprise light signal generator(s) (e.g. LEDs) wherein the outputs of the apparatus comprise light signals (e.g. LED light signals).

[0070] The processor may be configured to determine a user-intended pointing location on the user-facing surface of the apparatus based at least on the estimated zenith angle and estimated azimuthal angle of the conductive object.

[0071] The processor may be configured to generate a control signal for the apparatus based at least on the determined user-intended pointing location.

[0072] The system may comprise any of the features, combinations of features and / or sub-combinations of features described herein.

[0073] Another aspect of the invention provides a method for controlling an apparatus operable with a touchless interface. The method comprises: receiving one or more capacitive sensor signals from one or more capacitive sensors, the one or more capacitive sensors sensitive to disturbances in their electric fields caused by a conductive object in proximity thereto; generating, based on the one or more capacitive sensor signals, a plurality of time-sequenced single reference points, each single reference point comprising a location of a center of mass of the conductive object; generating one or more machine learning inputs based on the plurality of time-sequenced single reference points; feeding the one or more machine learning inputs into a trained machine learning algorithm to thereby generate (e.g. infer) one or more estimation parameters, wherein the one or more estimation parameters comprise at least an estimated zenith angle with which the conductive object is oriented relative to an interaction surface of the interface; estimating a user intended pointing location of the conductive object based on the one or more estimation parameters; generating one or more control parameters, wherein the one or more control parameters comprise at least the user intended pointing location; and, generating a control signal for the apparatus based at least in part on the one or more control parameters.

[0074] Another aspect of the invention provides a method for controlling an apparatus operable with a touchless interface. The method comprises: receiving one or more capacitive sensor signals from one or more capacitive sensors, the one or more capacitive sensors sensitive to disturbances in their electric fields caused by a conductive object in proximity thereto; generating, based on the one or more capacitive sensor signals, a plurality of time-sequenced single reference points, each single reference point comprising a location of a center of mass of the conductiveobject; generating one or more machine learning inputs based on the plurality of time-sequenced single reference points; feeding the one or more machine learning inputs into a trained machine learning algorithm to thereby generate (e.g. infer) one or more estimation parameters, wherein the one or more estimation parameters comprise at least an estimated zenith angle with which the conductive object is oriented relative to an interaction surface of the interface; and estimating a user intended pointing location of the conductive object based on the one or more estimation parameters.

[0075] The one or more machine learning inputs may comprise at least one of: at least a portion of the plurality of time-sequenced single reference points; an estimated trajectory of the conductive object; velocity values of the conductive object; acceleration values of the conductive object; and, an azimuthal angle of the conductive object about an axis normal to the interaction surface.

[0076] The acceleration values may be used to identify stopping motions and / or tapping motions and the at least a portion of the plurality of time-sequenced single reference points comprises time-sequenced single reference points corresponding to the stopping motions and / or tapping motions.

[0077] The method may comprise determining a reliability score of the plurality of time-sequenced single reference points based on the velocity values.

[0078] Feeding the one or more machine learning inputs into the trained machine learning algorithm may comprise weighting the one or more machine learning inputs based on the reliability score.

[0079] The one or more estimation parameters may comprise an estimate of an azimuthal angle of the conductive object about an axis normal to the interaction surface.

[0080] The estimated trajectory may be estimated based at least on the plurality of time-sequenced single reference points.

[0081] The plurality of time-sequenced single reference points may be detected within a time window. The control signal may be generated in response to signals detected in the time window.

[0082] The time window may comprise a rolling time window with a start time updated in real time (e.g. for each single reference point).

[0083] Training data for training the machine learning algorithm may comprise robotic arm training data generated by a robotic arm with joints and end-effectors to simulate a human hand and fingers.

[0084] The robotic arm may be programmed to simulate human motions for interacting with a near-apparatus touchless interface.

[0085] The robotic arm may be programmed to generate varied training data by randomizing a simulated trajectory of the motions between a starting position and an ending position.

[0086] The simulated human motions may comprise one or more of: swiping, finger twirling, tapping, touching, finger hovering, finger dragging.

[0087] Training data for the machine-learning algorithm may comprise one or more of: time-sequenced single reference points comprising location of a center of mass of portions of the robotic arm within a detection range of the capacitive sensors; velocity values of the robotic arm; acceleration values of the robotic arm; an azimuthal angle of the robotic arm about an axis normal to the interaction surface.

[0088] Training data for training the machine learning algorithm may comprise unrealistic motions categorized as invalid gestures and the machine learning algorithm is trained to identify the invalid gestures as non-user-intended signals.

[0089] The method may comprise providing a loss function to optimize trainable parameters of the machine learning algorithm. The loss function may include term(s) that assign cost to at least one of: a difference in estimated values of training estimation parameters generated by the machine learning algorithm and actual (e.g. ground truth) values of training estimation parameters; and, a difference in the estimated pointing location and an actual (e.g. ground truth) user-intended location.

[0090] Training data for training the machine learning algorithm may comprise real user data for specified gestures.

[0091] The specified gestures may comprise a finger tapping motion and / or a stopping motion.

[0092] Training data for training the machine learning algorithm comprise simulated tapping motions labelled as tapping motions and simulated non-tapping motions labelled as non-tapping motions and providing a loss function to optimize trainable parameters of the machine learning algorithm wherein the loss function includes a term that assigns cost to misidentifying tapping motions and non-tapping motions.

[0093] Another aspect of the invention provides a system for receiving input to control an apparatus operable with a touchless interface. The system comprises: one or more capacitive sensors which are sensitive to disturbances in their electric fields caused by a conductive object in proximity thereto; a processor connected to receiveone or more capacitive sensor signals from the one or more capacitive sensors. The processor configured to: generate a plurality of time-sequenced single reference points based on the one or more capacitive sensor signals, each single reference point comprising a location of a center of mass of the conductive object; generate one or more machine learning inputs based on the plurality of time-sequenced single reference points; generate (e.g. infer), using a trained machine learning algorithm and the one or more machine learning inputs, one or more estimation parameters, wherein the one or more estimation parameters comprises at least an estimated zenith angle with which the conductive object is oriented relative to an interaction surface of the interface; estimate a user intended pointing location of the conductive object based on the one or more estimation parameters; generate one or more control parameters wherein the one or more control parameters comprise at least the user intended pointing location; and, generate a control signal to control the apparatus based on the one or more control parameters.

[0094] Another aspect of the invention provides a system for receiving input to control an apparatus operable with a touchless interface. The system comprises: one or more capacitive sensors which are sensitive to disturbances in their electric fields caused by a conductive object in proximity thereto; a processor connected to receive one or more capacitive sensor signals from the one or more capacitive sensors. The processor configured to: generate a plurality of time-sequenced single reference points based on the one or more capacitive sensor signals, each single reference point comprising a location of a center of mass of the conductive object; generate one or more machine learning inputs based on the plurality of time-sequenced single reference points; generate (e.g. infer), using a trained machine learning algorithm and the one or more machine learning inputs, one or more estimation parameters, wherein the one or more estimation parameters comprises at least an estimated zenith angle with which the conductive object is oriented relative to an interaction surface of the interface; estimate a user intended pointing location of the conductive object based on the one or more estimation parameters.

[0095] The system may comprise a protective layer overlaid on the capacitive sensors.

[0096] Contact between the conductive object and a user-facing surface of the protective layer may be recorded as touch data and the one or more machine learning inputs comprise the touch data when the touch data are available.

[0097] The contact between the conductive object and the user-facing surface of the protective layer may be a result of a tapping motion.

[0098] The protective layer may comprise optically transparent material.

[0099] The protective layer may be made of fiber optic glass. The apparatus may comprise a display configured to display images wherein the displayed images are optically transmitted by the fiber optic glass to a user-interactive surface of the protective layer.

[0100] The system may comprise any of the features, combinations of features and / or sub-combinations of features described above. Such method steps may be performed by a suitably configured processor.

[0101] Another aspect of the invention provides apparatus having any new and inventive feature, combination of features, or sub-combination of features as described herein.

[0102] Another aspect of the invention provides methods having any new and inventive steps, acts, combination of steps and / or acts or sub-combination of steps and / or acts as described herein.

[0103] It is emphasized that the invention relates to all combinations of the above features, even if these are recited in different claims.

[0104] Further aspects and example embodiments are illustrated in the accompanying drawings and / or described in the following description.Brief Description of the Drawings

[0105] The accompanying drawings illustrate non-limiting example embodiments of the invention.

[0106] FIG. 1 A is perspective view of a schematic illustration showing an example interaction with a near-apparatus touchless interface system.

[0107] FIG. 1B is plane view of a schematic illustration showing another example interaction with a near-apparatus touchless interface system.

[0108] FIG. 2 is flowchart of a method for estimating an orientation of a conductive object and / or a user-intended pointing location for interacting with an apparatus operable by a touchless interface system according to an example embodiment.

[0109] FIG. 3A is a schematic time-sequenced illustration of a conductive object approaching an apparatus operable by a touchless interface system with velocity according to an example embodiment.

[0110] FIG. 3B is a schematic time-sequenced illustration of a conductive objectapproaching an apparatus operable by a touchless interface system with acceleration (deceleration) according to an example embodiment.

[0111] FIG. 3C is a schematic time-sequenced illustration of an estimated trajectory of a conductive object approaching an apparatus operable by a touchless interface system according to an example embodiment.

[0112] FIG. 3D is a schematic time-sequenced illustration of an estimated azimuthal angle of a conductive object according to an example embodiment.

[0113] FIG. 3E is a schematic diagram showing a tapping motion of a conductive object according to an example embodiment.

[0114] FIG. 4 is a perspective view of a schematic illustration of a near-apparatus touchless interface system with a protective layer according to an example embodiment.Detailed Description

[0115] Throughout the following description, specific details are set forth in order to provide a more thorough understanding of the invention. However, the invention may be practiced without these particulars. In other instances, well known elements have not been shown or described in detail to avoid unnecessarily obscuring the invention. Accordingly, the specification and drawings are to be regarded in an illustrative, rather than a restrictive sense.

[0116] Systems and methods for estimating an orientation and / or a user-intended pointing location for interacting with an apparatus operable with a touchless interface are disclosed herein. Systems and methods disclosed herein address the problem of potential misalignment between a user-intended pointing location and an estimated pointing location in near-apparatus touchless interface systems by estimating the pointing location based at least on an estimated zenith angle and an estimated azimuthal angle of a conductive object interacting with the touchless interface. In particular, there is a general correlation between a zenith angle of the conductive object measured from a reference zenith axis extending perpendicular to a userfacing surface of the apparatus and the magnitude of discrepancy (i.e. distortion) between the user-intended pointing location and the estimated pointing location.

[0117] A conductive object is tracked by one or more capacitive sensors within a detection range of the capacitive sensors to generate a plurality of time-sequenced single reference center of mass. Each single reference center of mass point corresponds to a location of a center of mass of the conductive object as a singlereference point and has an associated time stamp. One or more machine learning inputs are generated based on the plurality of time-sequenced single reference center of mass points. The one or more machine learning inputs are provided to a trained machine learning algorithm that, when executed, generates one or more estimation parameters. The one or more estimation parameters comprise at least an estimated zenith angle and an estimated azimuthal angle of the conductive object. The userintended pointing location is then estimated based on the estimation parameters. A control signal for the apparatus may be generated based at least on the estimated pointing location of the conductive object.

[0118] FIG. 1A is a perspective view of a schematic illustration 10A showing an example user interaction with a touchless interface 17. In FIG. 1 A, a conductive object 15 (which is shown as a human hand in FIG. 1A) is in proximity to a display 14 of an apparatus including a near-apparatus touchless interface 17. Conductive object 15 is attempting to target (e.g. point at) an indicium 19 (e.g. a button) shown on display 14. In FIG. 1A, display 14 extends in a plane defined by the %-axis and the y-axis. Capacitive sensors (not illustrated in FIG. 1 A) of near-apparatus touchless interface 17 detect a location of a center of mass of conductive object 15 as a single reference single reference center of mass point 27. A simple projection of single reference center of mass point 27 in a z-direction parallel with the z-axis onto display 14 results in a projected center of mass location 13. However, as can be seen from the FIG. 1 A illustration, projected center of mass location 13 is spaced apart from user-intended location 11 at least along the y-axis. Therefore, simply taking projected center of mass location 13 as the user-intended pointing location would result in a discrepancy between user-intended location 11 and projected center of mass location 13. Such discrepancy may cause misalignment between the user intention and the response from the apparatus.

[0119] The discrepancy between projected center of mass location 13 and userintended location 11 is related, at least in part, to a zenith angle 0 of conductive object 15 measured from a zenith reference axis 18 extending perpendicularly to surface of touchless interface 17 (i.e. parallel to the z-axis) extending through user-intended location 11. In some embodiments, the near-apparatus touchless interface system comprises a motion sensor (e.g. inertial measurement unit (IMU), accelerometer, etc.) configured to measure a motion (i.e. orientation, vibration, etc.) of the apparatus to thereby ascertain an orientation of the capacitive sensors / display of the apparatus orvibratory motions of the apparatus to thereby derive accurate reference axes and / or tune out noise from vibratory motions. In FIG. 1 A, the user intends for a distal portion 15A (e.g. fingertip) of conductive object 15 to indicate the user’s intention of pointing at user-intended location 11. However, capacitive sensors of near-apparatus touchless interface 17 detects conductive object as a single reference point 27 at a location of the center of mass of the entire conductive object 15 within the detection range of the conductive sensors. The relatively large mass of the portions of the user’s hand other than the fingertip result in single reference point 27 being projected to projected center of mass location 13 along the z-direction.

[0120] As the discrepancy between projected center of mass location 13 and userintended location 11 shown in FIG. 1 A demonstrates, any near-apparatus touchless interface that operate by detecting a center of mass of a conductive object as a single reference point may be prone to alignment errors due to presence of additional conductive mass within a detection range of the sensors of the near-apparatus touchless interface.

[0121] FIG. 1B is a plane view of a schematic illustration 10B showing another example interaction between a conductive object and a near-apparatus touchless interface system. In FIG. 1B, conductive object 15 comprises a human arm extending from a shoulder 31 to a wrist joint 29 as well as a human hand extending from wrist joint 29 to a fingertip 15A. A vector 33 can be drawn from wrist joint 29 to fingertip 15A and may be referred to as the “wrist-to-finger vector”. Similarly, a vector 37 can be drawn from shoulder 31 to wrist joint 29 and may be referred to as the “shoulder-to-wrist vector”.

[0122] As can be seen in FIG. 1 B, both wrist-to-finger vector 33 and shoulder-to-wrist vector 37 may be relevant for determining an orientation of the conductive object within detection range of a touchless interface to thereby ascertain user intention. Specifically, an azimuthal angle <> measured from an azimuthal reference axis extending in parallel to the surface 14 of the touchless interface is important for ascertaining an orientation of conductive object 15.

[0123] As illustrated in FIGs. 1 A and 1 B, it may be desirable to use information relating to one or more of zenith angle 0, azimuthal angle / >, wrist-to-finger vector 33, and shoulder-to-wrist vector 37 to achieve accurate determination of an orientation of conductive object 15 within detection range of touchless interface 17. If a user’s finger / hand is perpendicular to a surface of interaction (i.e. , parallel to the z-axis), thedetected location of selection point (e.g. point 13) may be relatively aligned (i.e., accurate) with the user-intended location (e.g. user-intended location 11) because the zenith angle 0 is negligibly small and the effect of wrist-to-finger vector 33 and shoulder-to-wrist vector 37 may also be negligible. However, as the magnitude of zenith angle 0 increases, more mass of the conductive object may enter the detection range of the capacitive sensors. As a result, the single reference single reference center of mass point of the conductive object 15 within the detection range (e.g. point 27) may shift, which may lead to discrepancy between the user’s intended location (e.g. user-intended location 11) and the system’s detected location (e.g. point 13).

[0124] The discrepancy generally becomes more pronounced as 0 approaches 90 degrees (i.e., when the conductive object approaches an orientation that is generally parallel to detection plane (e.g. screen 14) of the interface apparatus 17).

[0125] FIG. 2 is flowchart of a method 100 for estimating an orientation of a conductive object and / or a user-intended pointing location for interacting with an apparatus operable by a touchless interface according to an example embodiment. Method 100 may be performed by any suitable processor(s). The processors may be embedded in the touchless interface system or in data communication with thereof. In FIG. 2, a processor 120 is shown schematically to perform one or more steps of method 100.

[0126] Method 100 begins with block 103 which comprises the step of continuously measuring a center of mass of a conductive object within a detection range of capacitive sensors 110 as a single reference point (referred to as “single reference center of mass points” herein) in real-time. In some embodiments, the single reference center of mass points 110 are continuously measured over successive time windows. In some embodiments, each time window is of a predetermined / configurable time duration. In some embodiments, the duration of the time window is selected at least on the basis of an operating frequency of capacitive sensors 110. The single reference center of mass points of the conductive object may be measured by any suitable capacitive sensors 110. In some embodiments, at least one electrode of each of capacitive sensors 110 is located on a user-facing side of an apparatus 140. In some embodiments, at least one electrode of each of the capacitive sensors 110 is fabricated from transparent conductive material, so that a human user can see a display of apparatus 140 through the at least one electrode of each of the capacitive sensors 110.

[0127] By performing block 103 over successive time windows, capacitive sensors 110 are able to measure a plurality of single reference center of mass points 106 of the conductive object in each time window where each single reference center of mass point is discretely spaced apart in time from the other single reference center of mass points. An example center of mass 106 at least comprises information about the position of the single reference center of mass points of the conductive object and the associated time stamp: (x, y, z, t), where x is the x-position of the single reference center of mass point along an x-axis, y is the y-position of the single reference center of mass point along a y-axis, z is the z-position of the single reference center of mass point along a z-axis, where the x-, y- and z-axes are mutually orthogonal, and t is the corresponding time stamp. In other words, block 103 measures a plurality of time-sequenced single reference center of mass points 106 of the conductive object. In some embodiments, single reference center of mass points 106 are measured at predetermined / configurable regular time intervals, i.e. , the time duration between any two sequential pair of time stamps is constant.

[0128] After measuring a plurality of single reference center of mass points 106 in a corresponding time window, method 100 proceeds to block 107 which comprises the step of generating machine learning inputs 108 (“ML inputs 108”) from the plurality of single reference center of mass points 106 in the given time window. Processor 120 is configured to perform the step of generating ML inputs 108 in block 107. In some embodiments, ML inputs 108 comprise at least the plurality of single reference center of mass points 106.

[0129] In some embodiments, ML inputs 108 comprise one or more average velocity values of conductive object calculated from the plurality of single reference center of mass points 106. FIG. 3A is a schematic time-sequenced illustration 200A of a conductive object 250 approaching a display with velocity according to an example embodiment. In FIG. 3A, as conductive object 250 approaches the display (not illustrated in FIG. 3A), a series of single reference center of mass points 227-1 to 227-4 (collectively, “single reference center of mass points 227”) is detected and recorded along with their respective time stamps. The value of the average velocity of the conductive object 250 between any pair of single reference center of mass points 227 can be approximated as the difference in the position between the two single reference center of mass points 227 divided by the time duration between the corresponding time stamps. For example, an average velocity of conductive object250 between single reference center of mass points 227-1 and 227-2 can be calculated as the difference in displacement between single reference center of mass points 227-1 and 227-2 divided by the difference in time between single reference center of mass points 227-1 and 227-2. In some embodiments, the average velocity calculated from any two single reference center of mass points 227 is associated with a velocity time stamp. In some embodiments, the velocity time stamp is selected as the midpoint between the corresponding time stamps of the single reference center of mass points 227 used to calculate the average velocity.

[0130] In some embodiments, ML inputs 108 comprise one or more approximated acceleration values (or deceleration values) of conductive object calculated from the plurality of single reference center of mass points 106. FIG. 3B is a schematic time-sequenced illustration 200B of a conductive object 250 approaching a display while decelerating according to an example embodiment. In FIG. 3B, as conductive object approaches the display (not illustrated in FIG. 3B), a series of single reference center of mass points 227-5 to 227-9 (collectively, “single reference center of mass points 227”) is detected and recorded along with their respective time stamps. The approximate acceleration value of the conductive object 250 between any two average velocity values calculated as described elsewhere herein can be approximated as the difference in the average velocities divided by the difference in associated velocity time stamps. For example, average velocity values of conductive object 250 between single reference center of mass points 227-1 and 227-2 and between points 227-2 and 227-3 can be calculated by the method discussed above. The average velocity between single reference center of mass points 227-1 and 227-2 is associated with a first velocity time stamp being the midpoint between the respective time stamps of single reference center of mass points 227-1 and 227-2 and the average velocity between single reference center of mass points 227-2 and 227-3 is associated with a second velocity time stamp being the midpoint between the respective time stamps of single reference center of mass points 227-2 and 227-3. Then, an approximate acceleration value of conductive object 250 between the first and second velocity time stamps can be calculated as the difference in the average velocity values divided by the corresponding time difference between the first and second velocity time stamps.

[0131] In some embodiments, ML inputs 108 also comprise a vector value representing an estimated trajectory of the conductive object within a correspondingtime window. FIG. 3C is a schematic time-sequenced illustration 200C of a conductive object 250 approaching an apparatus 140 with an estimated trajectory 219 according to an example embodiment. As shown in FIG. 30, a trajectory 219 can be estimated by tracing a series of single reference center of mass points 227-10 to 227-13 recorded in a given time window in chronological order. Various techniques for estimating a trajectory based on time-sequence single points are applicable and, for the sake of brevity, will not be elaborated upon in this application. It is to be understood any suitable trajectory estimating techniques may be applied to obtain estimated trajectory 219 based on single reference center of mass points 227. In some embodiments, an orientation of trajectory 219 can be determined, for example, by calculating a slope of trajectory 219. In some embodiments, trajectory 219 is estimated only when one or more pre-determined conditions are met. In one nonlimiting example embodiment, a condition for estimating trajectory 219 is when the value of a z-position of at least one single reference center of mass point in a time window crosses below a threshold z-position value, which may indicate an intention from a user to engage with the near-apparatus touchless interface. In another nonlimiting example embodiment, another condition for estimating trajectory 219 is when the difference in the z-position between at least two single reference center of mass points in a time window exceed a threshold magnitude signaling non-trivial movement of a user (e.g. a hand of user) relative to the near-apparatus touchless surface along the z-axis, which may indicate an intention from the user to either engage or disengage with the near-apparatus touchless surface.

[0132] In some embodiments, a predetermined and / or configurable threshold value for the number of single reference center of mass points collected within a given time window may be required for calculating an estimated trajectory. In other words, if the number of single reference center of mass points collected within a given time window exceeds the threshold number, an estimated trajectory will be calculated. If the number of single reference center of mass points is below the threshold number, then the calculation of the estimated trajectory may be omitted.

[0133] In some embodiments, ML inputs 108 comprise orientation angle of the near-apparatus touchless interface and / or the orientation angle of a display of the apparatus. The orientation angle of the near-apparatus touchless interface and / or the orientation angle of a display of the apparatus may be measured and provided by motion sensor(s) (e.g. IMU sensors, accelerometers, etc.). The orientation of thenear-apparatus touchless interface and / or the orientation angle of a display of the apparatus typically have direct influence on the orientation of the conductive object and may serve as important parameters in estimating the zenith angle of the conductive object. For example, typically, if the display of the apparatus is oriented such that the display is substantially perpendicular to a line of sight of the user, the expected zenith angle of the conductive object relative to the display of the apparatus is likely relatively small. On the other hand, if the display of the apparatus is oriented such that the display is oriented such that the display is relatively less perpendicular to the line of sight of the user, the expected zenith angle of the conductive object relative to the display of the apparatus is likely relatively larger.

[0134] In some embodiments, average velocity values are used to evaluate reliability in the calculations for a particular set of single reference center of mass points. For example, the faster the conductive object moves, the greater the uncertainty in parameter values calculated from the corresponding single reference center of mass points. Therefore, if the calculated average velocity value exceeds a predetermined and / or configurable threshold velocity value , then the uncertainty in the parameter values calculated from the corresponding set of single reference center of mass points is likely higher compared to the uncertainty in the parameter values calculated from a set of single reference center of mass points where the calculated average velocity value is below the threshold velocity value.

[0135] The reliability (e.g. uncertainty in the calculated parameter values) may be associated with the corresponding set of parameter values and be a factor in the utilization of the parameter values as ML inputs 108. For example, datasets and associated parameter values with relatively higher uncertainty (i.e. lower reliability scores) may be given comparatively lower weight in ML inputs 108 compared to datasets and associated parameters with relatively lower uncertainty (i.e. higher reliability scores).

[0136] Approximated acceleration values may be particularly relevant in detecting “stopping motions”. “Stopping motions” as described herein refer to a motion of the conductive object during which a velocity of the conductive object decreases to substantially zero, as determined based on one or more estimated acceleration values. A non-limiting example stopping motion is when a hand of a user with an extended pointing finger approaches the user-interactive surface of the apparatus and then coming to a stop as the user sees that the near-apparatus touchless interfacehas detected the presence of the user’s hand and has generated an indicia on the apparatus accordingly. The sequence of motion illustrated in FIG. 3C is representative of a stopping motion. As conductive object 250 approaches userintended location 211 and is detected as points 227-10 to 227-13, conductive object 250 decelerates until coming to approximately a stop. This characteristic of deceleration is a key signature to stopping motions and therefore approximated acceleration values may be particularly important for identifying stopping motions.

[0137] In some embodiments, when stopping motions are identified via the approximated acceleration values, only time-sequenced single reference center of mass points corresponding to the stopping motions are selected as machine learning inputs 108 for the corresponding time window. Typically, when a conductive object comes to a momentary stop after going through stopping motions, the distal portion of the conductive object is most likely to be substantially aligned with a general longitudinal extension of the conductive object. Therefore, the time-sequenced single reference center of mass points corresponding to the stopping motions are most relevant to the determination of a relevant zenith angle of the conductive object. Therefore, in some embodiments, even if multiple motions are detected in a time window, if stopping motions are detected, block 107 generates machine learning inputs 108 primarily based on center of mas points corresponding to the stopping motions. For example, within a time window, a user hovers their hand over apparatus 140 within a detection range of capacitive sensors 110 while the user mulls over which virtual button to point at, and then extends their hand with the pointing finger towards the desired virtual button once the user makes a decision. The pointing motion goes through an initial acceleration and then into deceleration until coming to a stop (i.e. , a stopping motion). In this example, time-sequenced single reference center of mass points detected in response to the hovering motion may be discarded and only time-sequenced single reference center of mass points detected in response to the stopping motion (i.e. the pointing motion sequence) are kept because these single reference center of mass points corresponding to the stopping motion are the most relevant to determining the estimated orientation (i.e., zenith angle, azimuthal angle, etc.) of the conductive object.

[0138] In some embodiments, approximated acceleration values may also allow identification and consequently emphasis on datasets and parameters that are likely associated with tapping motions. FIG. 3E is a schematic diagram 200E showing atapping motion of a conductive object 250 according to an example embodiment. As shown in FIG. 3E, a conductive object 250 (shown as a human hand in FIG. 3E) utilizes its distal portion 250A (shown as a human finger in FIG. 3E) to indicate the user’s intention of interacting with user-intended location 211 on apparatus 140. The movement of distal portion 250A begins in an initial state 250A-1 and ends in a final state 250A-2. Distal portion 250A typically accelerates when starting from initial state 250A-1 and decelerates when reaching final state 250A-2. Therefore, approximated acceleration values may inform the likelihood of tapping motions and allow for emphasis on datasets and parameters associated with the specific tapping motions. In contrast, if a user is performing gestures such as moving their finger around generally in the x- and y-plane to draw, there is generally little acceleration along the z-axis.

[0139] As discussed above, near-apparatus touchless interfaces that incorporate capacitive sensors for detecting single reference center of mass points of the conductive object may be prone to alignment errors between a user-intended location and an estimated pointing location when additional conductive mass are also present within the detection range of the capacitive sensors. Therefore, presence of the additional conductive mass needs to be accounted for in the estimation of the user intended pointing location to reduce potential misalignment. In some embodiments, machine learning inputs 108 are generated from raw time-sequenced single reference center of mass points 106 for input into a machine learning algorithm to account for the presence of additional conductive mass in the estimation of user intended pointing location.

[0140] Returning to FIG. 2, after generating machine learning inputs 108 at block 107, method 100 proceeds to block 111 which comprises running a machine learning algorithm with machine learning inputs 108 to generate estimation parameters 112. The estimation parameters comprise at least an estimated zenith angle and an estimated azimuthal angle of the conductive object. In some embodiments, estimation parameters 112 also comprise orientation angle of the near-apparatus touchless interface and / or the orientation angle of a display of the apparatus. In some embodiments, processor 120 comprises a machine learning processor configured to run the machine learning algorithm.

[0141] FIG. 3D is a schematic time-sequenced illustration 200D of a conductive object 250 approaching a near-apparatus touchless surface at an estimatedazimuthal angle 229 according to an example embodiment. As shown in FIG. 3D, an azimuthal angle 229 is measured from an azimuthal reference line (e.g. line 235) extending in parallel to a surface of the touchless interface. In FIG. 3D, line 235 extends along the y-axis. This is not necessary. The azimuthal reference line may extend along any suitable axis that extends in the plane on which the touchless interface and / or the display of the apparatus extends. Although a user-intended location 211 is shown in FIG. 3D, user-intended location 211 is not required for determination of an estimated azimuthal angle 229. The inclusion of user-intended location 211 is merely to illustrate the relationship between trajectory 219, azimuthal angle 229 and user-intended location 211. In some embodiments, azimuthal angle 229 is an approximation of the finger-to-wrist angle. The “finger-to-wrist angle” as used herein refers to the angle measured from a reference axis (e.g. axis 235) of a human hand with an extending pointing finger. The finger-to-wrist angle is a useful metric in estimating user-intended location 211 because most human users tend to align their pointing finger with their wrist given the anatomy of a human hand.

[0142] As shown in FIG. 1 A, a zenith angle 0 is measured from a zenith reference axis extending orthogonally from an interactive surface of an apparatus (e.g. apparatus 140). The zenith angle 0 is highly relevant to the estimation of the orientation of the conductive object and consequently to user intention, because in a typical application setting, the larger the zenith angle 0, the greater the distortion to the single reference center of mass point due to the more proximal presence of additional conductive mass (e.g. hand portion 15B shown in FIG. 10) to the capacitive sensors.

[0143] In some embodiments, estimation parameters 112 comprise a shoulder-to-wrist vector (e.g. shoulder-to-wrist vector 37). The user’s shoulder is typically located outside a detection range of capacitive sensors 110. Nevertheless, a location of the user’s shoulder can still be estimated based on data collected over a plurality of time windows. Over the plurality of time windows, the respective zenith angles and the azimuthal angles are determined by the machine learning algorithm based on the machine learning inputs. Assuming a generally linear relationship between the wrist-to-shoulder angle and a location of the corresponding user-intended location on the apparatus for each interaction, the wrist-to-shoulder angle, and relatedly, the shoulder-to-wrist vector, may be estimated based on the plurality of zenith angles and azimuthal angles determined from the plurality of time windows. Therefore, over asufficient number of time windows, the shoulder location can be estimated. The estimated shoulder location and the associated shoulder-to-wrist vector and wrist-to-shoulder angle can then be provided as estimation parameters 112 for estimating user intended pointing location for subsequent time windows. The estimated shoulder location may be particularly useful as an estimation parameter in circumstances where the shoulder location of the user likely remains relatively constant across time windows.

[0144] The machine learning algorithm may be trained in any suitable manner on any suitable training data, comprising training the machine learning model with trainable parameters. In some embodiments, training the machine learning model comprises: initializing values for the trainable parameters; and performing a plurality of training iterations, each training iteration comprising: (i) determining, for each motion in a set of labelled motion training data, predicted azimuthal angle and predicted zenith angle based on current values for the trainable parameters of the machine learning model; (ii) calculating a loss value based on a comparison between the predicted azimuthal angle and predicted zenith angle and corresponding ground truth azimuthal angle and ground truth zenith angle from the labelled motion training data; (iii) computing gradients of the loss value with respect to the trainable parameters; and, (iv) updating the current values for the trainable parameters based on the computed gradients. In some embodiments, real user data for specified gestures are incorporated into the training data. In some embodiments, a robotic arm is utilized to generate training data. In some embodiments, imitation learning techniques are applied to train the robotic arm such that the robotic arm may generate simulated trajectories when given a starting position and an end target. The robotic arm, which includes joints, components and an end-effector to simulate a human’s hand and finger, dictates the ground-truth location of the user intended location (i.e. , corresponding to what a user would point to with the user’s fingertip). The robotic arm is thus able to generate a large quantity of high-fidelity data.

[0145] In some embodiments, the robotic arm can be set up to generate training data by programming the robotic arm to perform a variety of motion tasks simulating a conductive object (e.g. a human hand and finger) interacting with a near-apparatus touchless interface system and the machine learning algorithm is trained by receiving training inputs 108 to generate estimated estimation parameters 112. The estimation parameters 112 are then used to estimate a user-intended location. The differencebetween the estimated intended location and the true user-intended location (known from the setting of the robotic arm) is then determined to be used as inputs in a suitable loss function. The parameters of the machine learning algorithm may then be iteratively updated based on gradients of the loss function computed via backpropagation.

[0146] In some embodiments, a randomness factor is incorporated into generating the variety of motion tasks for the robotic arm to add variance to the training data. The training data motions include, but are not limited to, swiping, finger twirling, tapping, touching, finger hovering, finger dragging (at various angles), etc. Data inputs may include, but are not limited to, temporal (x, y, z) position data from the sensor, velocity and acceleration data calculated from the temporal position data, and standing angle of the sensor / screen (e.g., flat on table, flat on wall, or some angle in between). Additionally, to aid in applying the results of the model across different sensors (different sizes, materials, and form-factors), the use of eigenvalues and eigenvectors may be applied.

[0147] The robotic arm-generated training data may be routinely reviewed to identify anomalies and unrealistic motions to ensure that the training data is still relevant to real world use cases. Unrealistic motions may also be incorporated into the training data. For example, since unrealistic motions are unlikely to be user-intended gestures, unrealistic motions can be included in the training data so the machine learning algorithm can recognize them as invalid motions and reduce the potential for false-positives (i.e. the system identifying a motion as a user-intended motion when it is not). In some embodiments, invalid motions are categorized as unrealistic gesture inputs such that the machine learning algorithm can be trained to negate them in real world applications.

[0148] Any suitable positive and negative reinforcement techniques with policies aimed at rewarding the machine learning program for correct gesture recognition may be employed. A suitable loss function may be formulated to optimize the parameters of the machine learning algorithm. In some embodiments, the loss function includes terms that at least assigns cost based on the difference in estimated values of the estimation parameters and actual values of the estimation parameters and / or the difference in estimated user-intended location and the actual user-intended location. The robotic arm may be programmed to provide training data with specific motions. In some embodiments, the robotic arm is programmed to perform tapping motions andnon-tapping motions to train the algorithm to differentiate tapping motions from nontapping motions to thereby reduce false-positives of detecting a tapping motion in real world applications. The robotic arm may also be programmed to randomize its movement to better simulate the general variance in human hand motion. In some embodiments, the training data comprise datasets with simulated fixed-distance motions. Fixed-distance motions described herein refer to motions that occur at or approximately at a fixed distance from the capacitive sensors of the touchless interface for specific reasons. For example, if the apparatus operable by a touchless interface comprises a protective layer covering the capacitive sensors / display of the apparatus, then the conductive object is separated by a minimum distance from the capacitive sensors determined at least by a thickness of the protective layer, and potentially width of separation between the protective layer and the capacitive sensors (e.g. an air gap). An example fixed-distance motion is a dragging motion where a user moves its pointing finger relative to the capacitive sensors at a substantially fixed distance. The simulated fixed-distance motions comprise the endeffector of the robotic arm coming to a stop or moving relative to a user-facing side of the apparatus at fixed distances to thereby emulate motion of a conductive object at select distances relative to the capacitive sensors. Training the machine learning algorithm on datasets comprising the fixed-distance motions tunes the machine learning algorithm to identify fixed-distance motions of the conductive object as userintended actions and detect user interactions associated with the fixed-distance motions as eliciting a touch-like response from an interactive element on the userfacing side of the apparatus, for example, a “tapping” motion with the interactive element on the user-facing surface of the apparatus.

[0149] There are a number of advantages to applying a machine learning algorithm at block 111. First, each user has different physical attributes and different motion habits. A conventional algorithm is likely to have trade-offs between generalization and accuracy / precision due to each user’s specific characteristics. A machine learning algorithm, on the other hand, may be able to achieve better generalization while ensuring as much, if not more, accuracy / precision than conventional algorithms by tuning its parameters / weights to account for hidden variables and unknown permutations. Second, in many real life scenarios, there may be insufficient data due to a motion being too quick or too close to the capacitive sensors. When there are insufficient data, a conventional algorithm is severely constrained and prone tomistake and / or noise. On the other hand, a machine learning algorithm may be able to better contextualize the data and generate better estimates. Lastly, another benefit of using a ML algorithm instead of a conventional signal processing routine is the reduced latency of the ML algorithm. Gesture patterns and user’s motions can be well characterized by training data. Conventional signal processing routines often require various metrics to be met by the detected motion and generally require a large number of data points before it is able to reliably validate a detected gesture. On the other hand, ML algorithms are able to reliably validate detected gestures with less data due to the potential of trained pattern recognition.

[0150] In some embodiments, the machine learning algorithm is customizable to a user’s specific motion habits. For example, the user’s motion data can be tracked and stored and transformed into training data to further update the parameters / weights of the machine learning algorithm such that the machine learning algorithm can be tuned to improve both precision and accuracy in the estimation parameters 112 with respect to a user’s specific motion patterns / habits.

[0151] Returning to FIG. 2, after obtaining estimation parameters 112, method 100 proceeds to block 115 which estimating an orientation of the conductive object relative to an interactive surface of apparatus 140 and / or a user-intended pointing location based on estimation parameters 112. The estimated orientation and / or the estimated user-intended pointing location are determined based at least on the estimated zenith angle and the estimated azimuthal angle. The user-intended pointing location may comprise a pair of coordinates (x’, y’) where x’ is the x-coordinate of the pointing location along an x-axis and y’ is the y-coordinate of the pointing location along a y-axis where x- and y-axes together define a plane parallel to a user-facing surface of the apparatus.

[0152] After determining the estimated orientation of the conductive object and / or the estimated user-intended pointing location at block 115, method 100 may proceed to block 119 which comprises the optional step of generating a control signal for an apparatus 140 operable by the touchless interface based on control parameters 116 that comprise at least the estimated orientation of the conductive object (i.e., the estimated zenith angle and the estimated azimuthal angle) and / or the estimated userintended pointing location. In some embodiments, control parameters 116 also comprise the estimated trajectory (e.g. trajectory 219). Step of method 100 in block 119 is optional because a control signal may not be necessary when no user intentionfor interacting with apparatus 140 is detected based at least on the orientation of the conductive object as determined at block 115. Block 119 may be performed when user intention for interacting with apparatus 140 is detected.

[0153] For example, if method 100 detects a user moving their finger in an in-and-out motion towards apparatus 140 and the finger is proximal to a user-interactive (Ul) object (e.g., a virtual button) on apparatus 140, then method 100 could determine that the user’s intention is to select the virtual button based at least on the three variables discussed above. A suitable control signal 130 can then be generated to perform any actions associated with the selection of the virtual button and indicate to the user that the selection has been made. On the other hand, if method 100 detects a user generally hovering above apparatus 140 with relatively little movement in all directions, then method 100 could determine that there is no intention by from the user to elicit any specific response from apparatus 140.

[0154] In some embodiments, one or more algorithms are used to determine the nature of control signal 130. In a non-limiting example embodiment, one algorithm determines that a user intended to select a virtual button (based on the in-and-out motion) and another algorithm determines that the user intended pointing location sufficiently overlaps with the location of the virtual button on apparatus 140, which concludes that a valid virtual selection is made.

[0155] In another non-limiting example embodiment, block 119 determines the intention of a user to carry out a touchless gesture such as a swipe or finger twirling gesture. Since the user intended pointing location is now included among control parameters 116, touchless gestures such as swipes can be more accurately identified as the true intended motion as detected by capacitive sensors 110. For example, a swiping motion performed by a hand and extended finger with a large zenith angle 0 can skew the detection motion of the hand - e.g., the detected motion of the single reference center of mass points without correction / adjustment may not be as extended nor as linear as the user’s true intended motion.

[0156] With user intended pointing locations identified throughout the motion, which is at least based on the estimated zenith angle (which is continuously calculated and determined over successive time windows), the detection motion itself can be accurately determined so that control signal 130 can indicate a touchless gesture (e.g. swiping, finger twirling, hovering, etc.) that is recognized by apparatus 140.

[0157] Another type of user input that may be improved with method 100 and systemsapplying method 100 is ‘subtle touchless clicks’. Subtle touchless clicks refer to in / out motions whereby only the distal portion of the conductive object (e.g. the finger) moves and the remaining portion of the conductive object (e.g. the user’s hand) stays relatively still. In some cases, the finger may move 1cm or less towards the apparatus. Current conventional methods lack the ability to reliably detect subtle touchless clicks due to the minute motion and the inability to accurately identify the pointing location in situations where the single reference center of mass point is skewed by other conductive objects.

[0158] In a non-limiting example embodiment, a number of algorithms can work together to detect subtle touchless clicks. First, by the application of method 100, the zenith angle estimation algorithm can be used to determine the zenith angle and then the calibration algorithm can estimate a user intended pointing location. This allows for a much more accurate capture of the motion of the subtle touchless click (which typically consists of a small amount of change along the z-axis - i.e. , in / out of the screen). Another algorithm can then be separately trained to detect these minute motions in towards and out away from the screen to accurately characterize them as valid clicking motions instead of discarding these motions as sensor noise.

[0159] Block 119 of method 100 may be performed by any suitable algorithm by any suitable processors. In some embodiments, a machine learning algorithm is applied to generate control signal 130 based on control parameters 116. The machine learning algorithm may be trained in any suitable manner on any suitable training data. In some embodiments, a robotic arm can be set up to generate training data by programming the robotic arm to perform a variety of motion tasks simulating a conductive object (e.g. a human hand) interacting with a near-apparatus touchless interface system and the machine learning algorithm is trained by receiving training parameters 116 to generate control signals. A suitable reward (i.e. cost) function may be formulated to optimize the parameters of the machine learning algorithm. In some embodiments, the loss function includes terms that at least assign cost based on the difference in estimated pointing location and the actual pointing location and / or the difference in estimated user-intended action and the actual user-intended action.

[0160] Method 100 may be performed continuously in “real-time”, which means that Bock 103 of method 100 can be performed continuously such that single reference center of mass points 106 can be constantly updated from one time window to the next based on the motion of the conductive object as detected by capacitive sensors110. Block 107 can then be performed in real time to update machine learning inputs 108 based on the updated single reference center of mass points 106. Block 111 of method 100 can be run in real time to update estimation parameters 112 based on the updated machine learning inputs 108. Block 115 can be performed in real time to update control parameters 116 based on the updated estimation parameters 112. Optional block 119 can be performed in real-time based on updated control parameters 116 to generate control signal 130 in real time.

[0161] Systems and methods described herein improve the alignment between user intention and apparatus response for apparatus operable by near-apparatus touchless interface by reducing the discrepancy between user intention and estimation of user intention. Consequently, systems and methods described herein allow users to interact with an apparatus operable with a touchless interface with more freedom because motions and responses are highly aligned and do not need to be restricted to specific gestures. Systems and methods described herein also enable more optionality in deployment and instalment of the near-apparatus touchless interface since the alignment is easily adjustable based on the estimated zenith angle. Lastly, systems and methods described herein also enable the deployment of apparatuses with relatively larger interactive surfaces (e.g. larger screens) because systems and methods described herein are able to correct for distortions due to large angles (which is more likely to occur when a user interacts with apparatuses with larger interactive surfaces).

[0162] Further embodiments and technical features and advantageous implementations related to method 100 are described below.Protective Laver Overlaid on Capacitive Sensors

[0163] In some embodiments, a protective layer (i.e., glass, transparent or opaque plastic, wood, stone, etc.) is placed over the apparatus (e.g. a screen of the apparatus) to protect the apparatus from damage (e.g. accidental or vandalism). The use of protective layer has economic advantages since protective materials are typically less costly to replace compared to sensors of a touchless interface and / or displays of an apparatus in case of damage. In some embodiments, a protective layer with a thickness less than the detection range of the capacitive sensors is overlaid on top of the apparatus (e.g. on top of the screen / capacitive sensors of the apparatus). In some embodiments, the protective layer may be utilized as the user-interactivesurface of the apparatus.

[0164] FIG. 4 is a perspective view of a schematic illustration showing a touchless interface system 300 comprising a protective layer 329 overlaid on capacitive sensors 310 of the touchless interface system 300 according to an example embodiment. As can be seen in FIG. 4, a conductive object 315 (shown as a human hand in FIG. 4) is still detectable by capacitive sensors 310 within detection range 327 (illustrated as the volume enclosed by arcs 327 and a user-facing side of capacitive sensors 310 in FIG.4) of capacitive sensors 310 even though protective layer 329 is overlaid on capacitive sensors 310 to physically shield capacitive sensors 310 from conductive object 315. As shown in FIG. 4, conducive object 315 is able to indicate a userintended location 311 on protective layer 329. In some embodiments, protective layer 329 is made of optically transparent material (e.g. glass). In some embodiments, protective layer 329 is made of optically opaque material (e.g. wood). In some embodiments, the protective layer is spaced apart from a user-facing outer component of the apparatus (e.g. transparent capacitive sensors) by a separation distance (e.g. about 5mm) to define an air gap between the protective layer and the user-facing outer component of the apparatus. The air gap provides a buffer between the protective layer and the user-facing outer component of the apparatus to further protect the user-facing outer component of the apparatus. For example, even when the protective layer becomes somewhat deformed due to external force, the userfacing outer component of the apparatus is not impinged upon by the deformed protective layer due to the air gap in between. The air gap between the protective layer and the user-facing outer component also provides the benefit of heat insulation from external ambient heat (e.g. heat from sunlight) to protect the sensors (e.g. capacitive sensors) and / or components of the apparatus (e.g. a display of the apparatus) from overheating.

[0165] The protective layer may also facilitate hybrid touch / touchless interactions. For example, in a near-apparatus touchless interface with an optically transparent protective layer, the user can physically touch the surface of the protective layer and the sensors are detecting these interactions in a touchless manner. From a user’s perspective, there is tactile feedback, but the interaction is in fact facilitated by the near-apparatus touchless system. In a hybrid touch / touchless system, certain characteristics may be common to many hybrid touch / touchless interactions with the apparatus. For example, a user’s pointing finger can only extend as far as the outersurface of the protective layer when approaching the apparatus. Moreover, when the user’s pointing finger impinges upon the outer surface of the protective layer, there are likely local deformations of the tissues of the user’s fingertip due to the pressure from the impingement. These movement patterns / characteristics may have corresponding “signatures” in single reference center of mass points data. For example, user interactions that involve the user making contact with the outer surface of the protective layer may generally result in substantially similar z-coordinate values in the corresponding single reference center of mass points, because the user conductive object can only extend to the same z-axis position (i.e. the outer surface of the protective layer) when interacting with the apparatus in a hybrid touch / touchless mode and the z-axis position of the single reference center of mass points likely remain largely unchanged if the user interacts with the apparatus in a hybrid touch / touchless mode (e.g. swiping, dragging, etc.). As a result, the ML algorithm can be trained to develop ML parameters that take into account such signature patterns to thereby accurately identify user intention (e.g. pointing action, tapping action, swiping, etc.). Consequently, In some embodiments, the machine learning algorithm is trained on datasets including training data that comprise movement patterns of a conductive object interacting with a protective layer in a hybrid touch / touchless mode. By including example training data associated with hybrid touch / touchless interactions (e.g. a protective layer) in the training dataset, the ML algorithm can be trained to accurately identify hybrid touch / touchless user interactions.

[0166] In addition, since a user likely interacts with the protective layer as a touchbased surface while interacting with the apparatus via a touchless interface, the distortion introduced by the additional conductive mass is likely significant because the motion patterns are likely to be similar to motions commonly used for touch screens. Therefore, the incorporation of method 100 described herein is particularly advantageous to the embodiment of the apparatus comprising the protective layer, which is familiar to users (due to the use of touch interactions). Whether the user is selecting Ul objects, carrying out gestures, or simply drawing by moving a virtual cursor across the screen, an accurately estimated pointing location is essential to enhancing the system accuracy and improving user experience. As discussed elsewhere herein, an estimated pointing location for any near-apparatus touchless interface that operates by projecting a single reference center of mass point of a conductive object onto a user-facing surface of the apparatus may be prone toalignment errors due to presence of additional conductive mass within a detection range of the sensors of the near-apparatus touchless interface. Consequently, methods described herein may be applied to obtain an accurately estimated userintended pointing location, thereby correcting any potential misalignment between user intention and apparatus response as a result of simple projection of a single reference center of mass point onto the user-facing surface of the apparatus.

[0167] The same consideration above with respect to protective layers also applies to the use of gloves (particularly thick winter / work gloves or the like). For example, the gap between the screen and the user’s fingertip can cause distortions to the single reference center of mass points. Therefore, the application of method 100 described herein is also particularly advantageous to such an implementation.

[0168] In some embodiments, the protective layer is made of fiber optic glass or faceplates. In some embodiments, the fiber optic glass layer is provided as a lightweight layer of glass and placed on top of a display of an apparatus to transfer images shown on the display onto a user-facing surface of the fiber optic glass layer with high uniformity and color fidelity and minimal distortion. In some embodiments, the fiber optic glass layer is placed above or beneath sensors made of transparent conductive film material to provide protection to the display of an apparatus and also to allow the displayed images to be optically presented on the glass surface instead of underneath the protective layer on the original display, thus improving user experience.HoverTap™ Surface

[0169] In some embodiments, systems and methods described herein can be adapted to facilitating user interaction with an opaque surface (e.g. a table) in a user experience system to provide unique experiences. The user experience system may be adapted to / retrofitted to any existing opaque surface. In a non-limiting example embodiment, capacitive sensors are installed on a non-user-facing side of the opaque surface (e.g. underneath an opaque table) to measure single reference center of mass points of conductive objects (e.g. human hands) via capacitor signals as described elsewhere herein. An orientation of the conductive object can be estimated in real-time according to methods described herein to provide accurate identification of user intention. Responses to user intention (i.e. outputs from the user experience system) may be generated based on user intention ascertained by the systems andmethods described herein. The outputs of the system may respond to a variety of user movement, including, but not limited to, touchless interactions and / or hybrid touch / touchless interactions as described herein (e.g. user proximity, user gestures, tapping motions on the opaque surface, etc.). In some embodiments, the outputs from the user experience system are associated with the opaque surface. For example, light patterns can be projected from a light projector onto the opaque surface where the light projector receives inputs from the user experience system and generates light patterns that respond to the user movement as detected by the system capacitive sensors. In another example, light-emitting diodes (LEDs) or other types of light signal generators are embedded in the opaque surface programmed to be responsive to user motion relative to the opaque surface to output LED lights or other types of light signals. In some embodiments, the outputs from the user experience system are alternatively or additionally presented external to the opaque surface. For example, the user experience system is in data communication with an electronic device (e.g. a smartphone, computer, etc.) to generate signals to the electronic device where the electronic device generates outputs that respond to the user movement as detected by the system capacitive sensors. For example, a user may play a video game visually displayed on an electronic device by interacting with the opaque surface.Harmonious Touch and Touchless Applications

[0170] For near-apparatus interactions, the combination of touch and touchless input may be beneficial because not every motion benefits from being completely touchless. Some motions - particularly discrete interactions like changing a mode, confirming a selection, or pressing a virtual button - are best executed through direct touch, as it provides tactile feedback and reinforces user confidence. On the other hand, touchless gestures excel in scenarios where physical contact is inconvenient or unnecessary, such as quick shortcuts (e.g., swiping to go back a page) or analog adjustments (e.g., twirling a finger to adjust volume, hovering to move a slider, or freehand drawing without screen friction). Method 100 described herein creates a fluid user experience that balances precision, efficiency, and user comfort for harmonious touch and touchless applications.

[0171] To make such integration seamless, the ML algorithms described herein can be tuned to intelligently distinguish between touch and touchless interactions. Byanalyzing gesture patterns, hand positioning / angles, and user intention, the ML algorithms can ensure that both touch and touchless input methods are identified and executed seamlessly without interference. Additionally, multi-modal inputs, such as voice commands, can complement this system, allowing users to replace certain touch-based commands with speech (e.g., saying “set brightness to 10” instead of pressing a button) while still relying on touchless gestures for fine-tuned adjustments. This flexibility extends to Ul design, where different screen regions can be dedicated to specific input types — for example, touch-based buttons positioned along the edges for touch selections, while touchless sliders and rotary controls are placed in designated sections for gesture-based interactions.Touchless Multi-finger Detection

[0172] Although the capacitive sensors utilized and described herein detect only a single reference point (center-of-mass) at any given point in time, there are ways to implement multi-point touchless detection (e.g., for detecting multiple fingers).

[0173] A first method is a firmware solution which tunes the noise-sensitivity settings of the chipset used to collect sensor data and also the data pre-processing algorithms (running as firmware on the sensing device’s controller). By tuning the settings to allow the sensor to be more sensitive to touch input, it is possible to define a specific distance from the apparatus at which multiple fingers can be detected. However, the first method is generally limited to several mm above the interactive surface, not in the centimeters as it is typically for touchless gestures. This is possible because standard touch input has much higher sensor signal strengths than touchless input and if a very specific distance is known, then some flexibility in sensitivity is possible. This may be implemented if a specific material thickness is known and that material (e.g., glass, plastic, wood, etc.) is placed over top of the sensors. Alternatively, this may be achieved if a glove of specific thickness is used. The method generally works at the very specific distance from the apparatus and when the user touches the interactive surface.

[0174] The second method comprises a hardware solution which splits the sensors into multiple sensors (i.e., two halves, four quarters, or 6 / 8 / 9 / 10 sections, and so on). By utilizing multiple data collection chipsets and separated capacitive electrode sensor sheets to define separate electric fields, multiple sensors can be built onto the same sensor layer. While the second method is generally more expensive, it canallow for some granularity in detecting multiple fingers in a touchless manner. The simplest implementation is two halves which allows for two-finger gestures like pinch / zoom as long as the two fingers are placed over those two separate halves. A similar gesture could work if, for example, nine sensor sections are defined. To implement the second method, a signal processing routine continuously collates and analyzes the sensor data of all separate mini-sensors to detect if multiple fingers are being detect simultaneously. Then, the ML algorithms as described above could be utilized to detect any such multi-finger gestures (i.e., pinch / zoom) or swiping / twirling / hovering.Alternative Sensor Form-factors

[0175] The system described herein can be adapted into various form factors to meet the needs of different users and industries. In one non-limiting example embodiment, such capacitive sensors are integrated with transparent LCD screens. In another nonlimiting example embodiment, double-sided capacitive sensors are used to enable dual-sided interaction simultaneously, which allows users to engage with the apparatus from both the front and back, opening possibilities for applications such as interactive kiosks, real-time language translation applications, medical displays, or collaborative workstations where multiple users interact with the same apparatus simultaneously from different positions. In another non-limiting example embodiment, one sensor is ‘wrapped’ around the apparatus to allow a single user to use the apparatus one at a time (e.g., for language translation which occurs turn by turn). This implementation may be more economically cost effective to produce as it requires fewer number of chipsets.

[0176] Another non-limiting example approach is a partial sensor implementation, where only a section of the apparatus is equipped with the sensor for touchless sensing capabilities. This cost-efficient approach optimizes functionality by applying touchless interaction only where touchless interaction is most beneficial, such as in side panels for quick gestures, designated control zones for sliders or rotary inputs, or interactive overlays on top of an existing Ul without requiring full-screen coverage, etc. The adaptability of the systems described herein to different form factors ensures versatile adoptions across industries while balancing performance, usability, and cost efficiency.Multi-sensor Hardware Embodiment

[0177] In implementing the capacitive sensor described in this disclosure for humanmachine interfaces, tuning of the sensing electric-field can sometimes lead to scenarios where accuracy is higher closer to the apparatus and lower further from the apparatus. Alternatively, a specific detection range (e.g. 2-4cm) may be optimized at the cost of accuracy at other distances from the sensor. As a result, the ML algorithms described herein may not be able to receive sufficient data points to calculate the zenith angle reliably in some scenarios.

[0178] In one non-limiting example hardware embodiment, multiple coinciding sensors are integrated into a detection layer and each sensor is optimize for a specific detection range extending from the apparatus. The multi-sensor hardware embodiment uses multiple data-collection chipsets and multiple capacitive electrode patterns printed on a single user-facing side of the apparatus. In some embodiments, these multiple sensors are each laid onto separate layers of transparent conductive film / glass and overlap each other. In a non-limiting example embodiment, 4 distinct patterns are fabricated on a transparent conductive film laid on glass and each electrode pattern is connected to its own data collection chipset. Each sensor pattern is laid onto a separate layer of transparent conductive film. Each of these sensors have conjoining and expanding optimal detection ranges ranging from 0cm to the maximum (e.g., 12cm). For example: the 4 group of sensors respectively covering the detection range of: 0-2cm, 2-4cm, 4-8cm, and 8-12cm.

[0179] During operation, a signal processing routine may continuously cycle between the sensors (turning them on one after another turn by turn) to analyze each sensor’s data individually. Then, a ML algorithm described above can gather sufficient data points and track the trajectory of the conductive object at these varying detection ranges. The operating frequency of such a sensor embodiment may be more limited but a suitable number of sensors (i.e., 3-4) may facilitate a sufficiently high operating framerate for detecting the zenith angle during user interactions.Software Integration

[0180] To ease widespread adoption of such the systems described herein, a software development kit (SDK) - a comprehensive toolkit that includes an API for integrating touchless functionality into applications, may be developed. The SDK empowers developers and graphical user interface (GUI) designers by providing pre-built gesture templates, sample code, and tools that simplify the creation of intuitive user interfaces

[0181] For rapid and straightforward deployment, such systems may be offered with a USB-HID configuration, providing a plug-and-play solution that seamlessly integrates with existing systems. With this approach, the touchless sensor operates as a standard USB-HID device recognized by all modern PCs, ensuring compatibility with existing user interfaces without requiring software development. This simplicity allows customers to quickly integrate touchless functionality into their applications by mapping intuitive gestures — such as in-air clicks, swipes, or rotations — to familiar mouse and keyboard actions. With only minor adjustments to their current Ul, customers can access the benefits of touchless sensing. This integration method works hand-in-hand with all the ML algorithms described herein as those would operate on the firmware of such devices.

[0182] To further enhance usability, touchless sensors configured for USB-HID are supported by a Mobile App (available for Android and iOS), which enables customers to manage firmware updates, adjust gesture parameters, and customize interaction settings. This flexibility ensures that capacitive sensors can adapt to evolving user needs, providing a scalable and future-ready solution for any application.

[0183] The spectrum of solutions including USB-HID plug-and-play option and the SDK facilitate meeting diverse customer needs, from quick integration with minimal modifications to fully customized experiences. This dual approach ensures that touchless sensor technology remains accessible across industries while fostering a sustainable business model through hardware sales and tiered software licensing.

[0184] As industries adopt touchless interaction technologies for hygiene, durability, and enhanced usability, achieving response accuracy in near-apparatus touchless interfaces and enabling intuitive user experiences remain a significant challenge. Traditional far-range 3D gesture systems suffer from precision issues, user fatigue, and a learning curve, making them impractical for tasks requiring precise on-screen selections. Near-apparatus capacitive touchless interfaces like HoverTap™ offer a natural solution but face an alignment challenge: the touchless interface system detects a single reference point (typically the center of mass of the hand / finger), which introduces positional errors when the user’s pointing angle is not perpendicular to the screen.

[0185] Systems and methods disclosed herein address the commercial need forhighly accurate, natural touchless interactions with near-apparatus touchless interfaces by compensating for misalignments due to pointing angle. By determining the zenith angle of a user’s hand, the systems described herein can correctly align the indicia on the apparatus (e.g. cursor or touchless selection point) with the intended fingertip location, significantly improving accuracy and usability. This is particularly valuable in:• Medical environments, where surgical teams interact with screens through protective drapes and gloves.• Industrial and ruggedized environments, where operators wear thick gloves or interact with sealed displays.• Retail and self-service kiosks, where touchless navigation improves hygiene and accessibility.• Custom interfaces, where HoverTap is embedded in surfaces like wood, plastic, or glass overlays, enabling interactive surfaces beyond traditional touchscreens.

[0186] As touchless and hybrid interfaces gain widespread adoption, precision becomes a critical requirement for seamless user interaction and market acceptance. The systems and methods described herein enhance HoverTapTM’s commercial viability, ensuring touchless gestures and selections feel as natural and reliable as traditional touchscreen interactions.

[0187] The technology disclosed herein enhances HoverTapTM’s touch and touchless interfaces by improving accuracy when users interact at an angle. Systems and methods described herein ensure precise cursor alignment, touchless selection, and gesture recognition, even when barriers like gloves, protective glass, or overlays are present.

[0188] Key products benefiting from this invention include:• HoverTap™ Industrial Monitor - A hybrid touch / touchless display for medical, industrial, and rugged environments where gloves, drapes, or protective enclosures may be used.• HoverTap™ Rugged - Designed for outdoor and military applications, where thick protective glass shields the display from damage while users operate it with gloves. Without this invention, touchless clicks would misalign; with it, the touchscreen experience remains precise.• HoverTap™ Embedded Module - A sensor package for OEMs looking to addaccurate touchless interaction to their existing products, ensuring pointing angle compensation for more reliable gesture control.

[0189] By eliminating cursor drift and selection errors, the technology disclosed herein ensures HoverTap™ delivers a seamless and accurate touchscreen experience, even in challenging conditions.

[0190] The methods described herein may be embedded within the firmware of a compact sensor PCB, which is directly attached to the capacitive sensors in the touchless interfaces. This approach ensures efficient, low-latency processing of touch and touchless interactions, enabling accurate selection and gesture recognition even when users interact at an angle.

[0191] In industrial settings, flexibility in deployment is key. The technology disclosed herein can be practically applied through multiple product configurations:• OEM Integration (Board-Level Module): A barebones sensor and PCB solution that integrates directly with a customer’s existing monitor, touchscreen, or custom interface, allowing them to add high-precision touchless interaction while leveraging their own display hardware.• Standalone Industrial Monitor: A fully enclosed, commercial-grade monitor equipped with HoverTapTM’s sensor and firmware, ready for direct deployment in factories, clean rooms, and rugged industrial environments where touchscreens need to function through gloves, protective glass, or external barriers.• Sensor-Only Interface (No Screen): A sensor and PCB package that enables gesture-based interfaces without a traditional display. This is ideal for sealed control panels, industrial machinery, and kiosks, where physical buttons are replaced with static surfaces featuring capacitive sensing for touchless interaction.

[0192] By embedding the technology within the firmware of HoverTap™ sensor PCB, HoverTap™ may provide a scalable, adaptable solution that fits a range of industrial applications — whether customers need a full monitor, an embedded module, or a barebones sensor interface. This ensures HoverTap™ technology remains versatile, supporting both new system designs and retrofitting existing industrial equipment.

[0193] Key industries and use cases include, but are not limited to:• Medical & Healthcare - touchless interaction for surgical displays, radiology workstations, and bedside monitors, ensuring accuracy through gloves, steriledrapes, and protective glass.• Industrial & Manufacturing - hands-free operation of factory control panels, cleanroom interfaces, and automation systems, supporting gloved interaction and sealed environments.• Outdoor, Rugged & Military - reliable touchless interaction for military consoles, outdoor kiosks, transportation hubs, and ruggedized industrial equipment, where thick protective glass and gloves are used.• Retail & Self-Service Kiosks - hygienic, precise touchless controls for checkout systems, interactive kiosks, and customer-facing displays in high- traffic environments.• Custom Interfaces & Embedded Systems - integration into OEM products, such as touchless control panels, automotive interfaces, and interactive surfaces made from wood, plastic, or metal.

[0194] Several alternative methods could estimate a user’s pointing angle, but each comes with drawbacks in cost, complexity, or practicality:• 3D Camera Systems - depth-sensing cameras or LiDAR could track hand orientation, but 3D camera systems are expensive, computationally heavy, and require bulky hardware, making them impractical for embedded or industrial use cases.• Wearable Sensors - devices like gloves, rings, or wrist sensors could track orientation, but requiring users to wear extra hardware which limits adoption and also could not be applied to hands-free applications.• Multi-Sensor Capacitive Systems - using multiple capacitive sensors could improve tracking, but the multiple capacitive sensors typically interfere with each other’s electric fields and require solving physical placement, frequency modulation, and timing interference issues. This increases cost and complexity while adding calibration challenges.• Radar-Based Sensors - radar could theoretically infer pointing angles, but it’s unclear if it can achieve the necessary precision for fine touchless interactions.

[0195] While these methods may be feasible, they come with significant trade-offs. The technology disclosed herein provides a cost-effective, compact, and scalable solution using capacitive sensing, making it ideal for HoverTap™ and similar touchless interfaces.

[0196] Where a component (e.g. a software module, processor, assembly, device,circuit, etc.) is referred to herein, unless otherwise indicated, reference to that component (including a reference to a “means”) should be interpreted as including as equivalents of that component any component which performs the function of the described component (i.e. , that is functionally equivalent), including components which are not structurally equivalent to the disclosed structure which performs the function in the illustrated exemplary embodiments of the invention.

[0197] Embodiments of the invention may be implemented using specifically designed hardware, configurable hardware, programmable data processors configured by the provision of software (which may optionally comprise “firmware”) capable of executing on the data processors, special purpose computers or data processors that are specifically programmed, configured, or constructed to perform one or more steps in a method as explained in detail herein and / or combinations of two or more of these. Examples of specifically designed hardware are: logic circuits, application-specific integrated circuits (“ASICs”), large scale integrated circuits (“LSIs”), very large scale integrated circuits (“VLSIs”), and the like. Examples of configurable hardware are: one or more programmable logic devices such as programmable array logic (“PALs”), programmable logic arrays (“PLAs”), and field programmable gate arrays (“FPGAs”). Examples of programmable data processors are: microprocessors, digital signal processors (“DSPs”), embedded processors, graphics processors, math coprocessors, general purpose computers, server computers, cloud computers, mainframe computers, computer workstations, and the like. For example, one or more data processors in a control circuit for a device may implement methods as described herein by executing software instructions in a program memory accessible to the processors.

[0198] Processing may be centralized or distributed. Where processing is distributed, information including software and / or data may be kept centrally or distributed. Such information may be exchanged between different functional units by way of a communications network, such as a Local Area Network (LAN), Wide Area Network (WAN), or the Internet, wired or wireless data links, electromagnetic signals, or other data communication channel.

[0199] The invention may also be provided in the form of a program product. The program product may comprise any non-transitory medium which carries a set of computer-readable instructions which, when executed by a data processor, cause the data processor to execute a method of the invention. Program products according tothe invention may be in any of a wide variety of forms. The program product may comprise, for example, non-transitory media such as magnetic data storage media including floppy diskettes, hard disk drives, optical data storage media including CD ROMs, DVDs, electronic data storage media including ROMs, flash RAM, EPROMs, hardwired or preprogrammed chips (e.g., EEPROM semiconductor chips), nanotechnology memory, or the like. The computer-readable signals on the program product may optionally be compressed or encrypted.

[0200] In some embodiments, the invention may be implemented in software. For greater clarity, “software” includes any instructions executed on a processor, and may include (but is not limited to) firmware, resident software, microcode, code for configuring a configurable logic circuit, applications, apps, and the like. Both processing hardware and software may be centralized or distributed (or a combination thereof), in whole or in part, as known to those skilled in the art. For example, software and other modules may be accessible via local memory, via a network, via a browser or other application in a distributed computing context, or via other means suitable for the purposes described above.

[0201] Software and other modules may reside on servers, workstations, personal computers, tablet computers, and other devices suitable for the purposes described herein.Interpretation of Terms

[0202] Unless the context clearly requires otherwise, throughout the description and the claims:• “comprise”, “comprising”, and the like are to be construed in an inclusive sense, as opposed to an exclusive or exhaustive sense; that is to say, in the sense of “including, but not limited to”;• “connected”, “coupled”, or any variant thereof, means any connection or coupling, either direct or indirect, between two or more elements; the coupling or connection between the elements can be physical, logical, or a combination thereof;• “herein”, “above”, “below”, and words of similar import, when used to describe this specification, shall refer to this specification as a whole, and not to any particular portions of this specification;• “or”, in reference to a list of two or more items, covers all of the following interpretations of the word: any of the items in the list, all of the items in thelist, and any combination of the items in the list;• the singular forms “a”, “an”, and “the” also include the meaning of any appropriate plural forms. These terms (“a”, “an”, and “the”) mean one or more unless stated otherwise;• “and / or” is used to indicate one or both stated cases may occur, for example A and / or B includes both (A and B) and (A or B);• “approximately” when applied to a numerical value means the numerical value ± 10%;• where a feature is described as being “optional” or “optionally” present or described as being present “in some embodiments” it is intended that the present disclosure encompasses embodiments where that feature is present and other embodiments where that feature is not necessarily present and other embodiments where that feature is excluded. Further, where any combination of features is described in this application this statement is intended to serve as antecedent basis for the use of exclusive terminology such as "solely," "only" and the like in relation to the combination of features as well as the use of "negative" limitation(s)” to exclude the presence of other features; and• “first” and “second” are used for descriptive purposes and cannot be understood as indicating or implying relative importance or indicating the number of indicated technical features.

[0203] Words that indicate directions such as “vertical”, “transverse”, “horizontal”, “upward”, “downward”, “forward”, “backward”, “inward”, “outward”, “left”, “right”, “front”, “back”, “top”, “bottom”, “below”, “above”, “under”, and the like, used in this description and any accompanying claims (where present), depend on the specific orientation of the apparatus described and illustrated. The subject matter described herein may assume various alternative orientations. Accordingly, these directional terms are not strictly defined and should not be interpreted narrowly.

[0204] Where a range for a value is stated, the stated range includes all sub-ranges of the range. It is intended that the statement of a range supports the value being at an endpoint of the range as well as at any intervening value to the tenth of the unit of the lower limit of the range, as well as any subrange or sets of sub ranges of the range unless the context clearly dictates otherwise or any portion(s) of the stated range is specifically excluded. Where the stated range includes one or both endpointsof the range, ranges excluding either or both of those included endpoints are also included in the invention.

[0205] Certain numerical values described herein are preceded by "about". In this context, "about" provides literal support for the exact numerical value that it precedes, the exact numerical value ±5%, as well as all other numerical values that are near to or approximately equal to that numerical value. Unless otherwise indicated a particular numerical value is included in “about” a specifically recited numerical value where the particular numerical value provides the substantial equivalent of the specifically recited numerical value in the context in which the specifically recited numerical value is presented. For example, a statement that something has the numerical value of “about 10” is to be interpreted as: the set of statements:• in some embodiments the numerical value is 10;• in some embodiments the numerical value is in the range of 9.5 to 10.5;and if from the context the person of ordinary skill in the art would understand that values within a certain range are substantially equivalent to 10 because the values with the range would be understood to provide substantially the same result as the value 10 then “about 10” also includes:• in some embodiments the numerical value is in the range of C to D where C and D are respectively lower and upper endpoints of the range that encompasses all of those values that provide a substantial equivalent to the value 10

[0206] Specific examples of systems, methods and apparatus have been described herein for purposes of illustration. These are only examples. The technology provided herein can be applied to systems other than the example systems described above. Many alterations, modifications, additions, omissions, and permutations are possible within the practice of this invention. This invention includes variations on described embodiments that would be apparent to the skilled addressee, including variations obtained by: replacing features, elements and / or acts with equivalent features, elements and / or acts; mixing and matching of features, elements and / or acts from different embodiments; combining features, elements and / or acts from embodiments as described herein with features, elements and / or acts of other technology; and / or omitting combining features, elements and / or acts from described embodiments.

[0207] As will be apparent to those of skill in the art upon reading this disclosure, each of the individual embodiments described and illustrated herein has discrete components and features which may be readily separated from or combined with the features of any other described embodiment(s) without departing from the scope of the present invention.

[0208] Any aspects described above in reference to apparatus may also apply to methods and vice versa.

[0209] Any recited method can be carried out in the order of events recited or in any other order which is logically possible. For example, while processes or blocks are presented in a given order, alternative examples may perform routines having steps, or employ systems having blocks, in a different order, and some processes or blocks may be deleted, moved, added, subdivided, combined, and / or modified to provide alternative or sub-combinations. Each of these processes or blocks may be implemented in a variety of different ways. Also, while processes or blocks are at times shown as being performed in series, these processes or blocks may instead be performed in parallel, simultaneously or at different times.

[0210] Various features are described herein as being present in “some embodiments”. Such features are not mandatory and may not be present in all embodiments. Embodiments of the invention may include zero, any one or any combination of two or more of such features. All possible combinations of such features are contemplated by this disclosure even where such features are shown in different drawings and / or described in different sections or paragraphs. This is limited only to the extent that certain ones of such features are incompatible with other ones of such features in the sense that it would be impossible for a person of ordinary skill in the art to construct a practical embodiment that combines such incompatible features. Consequently, the description that “some embodiments” possess feature A and “some embodiments” possess feature B should be interpreted as an express indication that the inventors also contemplate embodiments which combine features A and B (unless the description states otherwise or features A and B are fundamentally incompatible).This is the case even if features A and B are illustrated in different drawings and / or mentioned in different paragraphs, sections or sentences.

[0211] It is therefore intended that the following appended claims and claims hereafter introduced are interpreted to include all such modifications, permutations, additions, omissions, and sub-combinations as may reasonably be inferred. Thescope of the claims should not be limited by the preferred embodiments set forth in the examples, but should be given the broadest interpretation consistent with the description as a whole.

Claims

WHAT IS CLAIMED IS:

1. A method for determining an orientation of a conductive object for interaction with an apparatus operable with a touchless interface, the method comprising:receiving one or more capacitive sensor signals from one or more capacitive sensors, the one or more capacitive sensors sensitive to disturbances in their electric fields caused by a conductive object in proximity thereto;determining, based on the one or more capacitive sensor signals, a plurality of time-sequenced single reference center of mass points, each single reference center of mass point corresponding to a location of a center of mass of the conductive object as a single reference point detected by the one or more capacitive sensors and associated with a time stamp;generating one or more machine learning inputs based on the plurality of time-sequenced single reference center of mass points; and,providing the one or more machine learning inputs to a trained machine learning algorithm comprising trainable parameters, that when executed, outputs an estimated zenith angle and an estimated azimuthal angle of the conductive object, wherein the estimated zenith angle is measured from a zenith reference axis extending perpendicularly to a user-facing surface of the apparatus and the estimated azimuthal angle is measured from an azimuthal reference axis extending in parallel to the user-facing surface of the apparatus.

2. The method according to claim 1 or any other claim herein wherein the one or more machine learning inputs comprise at least one of: at least a portion of the plurality of time-sequenced single reference center of mass points; an estimated trajectory of the conductive object; average velocity value(s) of the conductive object; and, estimated acceleration value(s) of the conductive object.

3. The method according to claim 2 or any other claim herein comprising determining the estimated trajectory based on the plurality of time-sequenced single reference center of mass points.

4. The method according to claim 3 or any other claim herein wherein determining the estimated trajectory comprises, at least in part, tracing the plurality oftime-sequenced single reference center of mass points in chronological order.

5. The method according to any one of claims 2 to 4 or any other claim herein comprising calculating the average velocity value between a pair of time-sequenced single reference center of mass points based on a distance between the pair of single reference center of mass points and a difference in the associated time stamps.

6. The method according to claim 5 or any other claim herein wherein each average velocity value is associated with a corresponding estimated velocity time stamp.

7. The method according to claim 6 or any other claim herein wherein the estimated velocity time stamp is determined as the temporal midpoint between the time stamps used to calculate the average velocity value.

8. The method according to any one of claims 6 to 7 or any other claim herein wherein the estimated acceleration value between a pair of average velocity values is calculated based on a difference between the pair of average velocity values and a difference between the associated estimated velocity time stamps.

9. The method according to claim 8 or any other claim herein wherein the machine learning algorithm is trained to identify stopping motions based at least on the estimated acceleration value(s), wherein the stopping motions comprise motions of the conductive object during which a velocity of the conductive object decreases to substantially zero.

10. The method according to any one of claims 1 to 9 or any other claim herein comprising continuously determining the plurality of time-sequenced single reference center of mass points within successive time windows, each time window of a predetermined and / or configurable time duration, to generate the one or more machine learning inputs associated with each corresponding time window.

11. The method according to claim 10 or any other claim herein wherein a starting time of each of the successive time windows is offset from a starting time of apreceding time window by a predetermined and / or configurable temporal offset.

12. The method according to any one of claims 1 to 11 or any other claim herein comprising training the machine learning model, wherein training the machine learning model comprises: initializing values for the trainable parameters; and performing a plurality of training iterations, each training iteration comprising:(i) determining, for each motion in a set of labelled motion training data, predicted azimuthal angle and predicted zenith angle based on current values for the trainable parameters of the machine learning model;(ii) calculating a loss value based on a comparison between the predicted azimuthal angle and predicted zenith angle and corresponding ground truth azimuthal angle and ground truth zenith angle from the labelled motion training data;(iii) computing gradients of the loss value with respect to the trainable parameters; and,(iv) updating the current values for the trainable parameters based on the computed gradients.

13. The method according to any one of claims 1 to 12 or any other claim herein wherein the motion training data for training the machine learning algorithm comprise training data generated by a robotic arm with joints and an end-effector to simulate human motions.

14. The method according to claim 13 or any other claim herein wherein the robotic arm is programmed to simulate human motions of interacting with a nearapparatus touchless interface.15 The method according to any one of claims 13 to 14 or any other claim herein wherein the robotic arm is programmed to generate varied training data by randomizing simulated trajectories of the end-effector.

16. The method according to any one of claims 13 to 15 or any other claim herein wherein the simulated human motions comprise one or more of: swiping, finger twirling, finger tapping, touching, finger hovering, finger dragging.

17. The method according to any one of claims 12 to 16 or any other claim herein wherein the motion training data comprise datasets with simulated fixed-distance motions, wherein the simulated fixed-distance motions comprises motions of the endeffector of the robotic arm stopping at or moving relative to a user-facing side of the apparatus at fixed distances to thereby emulate fixed-distance motions of a conductive object interacting with the user-facing side of the apparatus at fixed distances.

18. The method according to claim 17 or any other claim herein, wherein the machine learning algorithm is trained on datasets comprising the simulated fixed-distance motions to thereby develop the trainable parameters capable of identify fixed-distance motions of the conductive object.

19. The method according to claim 18 or any other claim herein comprising detecting user interactions associated with the fixed-distance motions as eliciting a touch-like response from an interactive element on the user-facing side of the apparatus.

20. A system for estimating an orientation of a conductive object for interaction with an apparatus operable with a touchless interface, the system comprising:one or more capacitive sensors which are sensitive to disturbances in their electric fields caused by a conductive object in proximity thereto;a processor connected to receive one or more capacitive sensor signals from the one or more capacitive sensors, the processor configured to:determine a plurality of time-sequenced single reference center of mass points based on the one or more capacitive sensor signals, each single reference center of mass point corresponding to a location of a center of mass of the conductive object as a single reference point and a corresponding time stamp;generate one or more machine learning inputs based on the plurality of time-sequenced single reference center of mass points;execute a trained machine learning algorithm with the one or more machine learning inputs to infer an estimated zenith angle and an estimated azimuthal angle of the conductive object,wherein the estimated zenith angle is measured from a zenith reference axis extending perpendicularly to a user-facing surface of the apparatus and the estimated azimuthal angle is measured from an azimuthal reference axis extending in parallel to the user-facing surface of the apparatus.

22. The system according to claim 21 or any other claim herein wherein the apparatus comprises a protective layer overlaid on the capacitive sensors.

23. The system according to claim 22 or any other claim herein wherein the protective layer is spaced apart from a user-facing side of the apparatus by a separation distance to define an air gap between the protective layer and the userfacing side of the apparatus to thereby protect the user-facing side of the apparatus from physical damage due to deformation of the protective layer (e.g. when an external force is applied against the protective layer) and / or to provide heat insulation to the user-facing side of the apparatus from ambient heat.

24. The system according to claim 22 or 23 or any other claim herein wherein motion training data for the machine learning algorithm comprise datasets with simulated motions emulating a conductive object coming into contact with the protective layer when interacting with the touchless interface to thereby train the machine learning algorithm to identify motion patterns specific to conductive object coming into contact with the protective layer.

25. The system according to any one of claims 22 to 24 or any other claim herein wherein the protective layer comprises optically transparent material.

26. The system according to claim 25 or any other claim herein wherein the protective layer is made of fiber optic glass and the apparatus comprises a display configured to display images wherein the displayed images are optically transmitted by the fiber optic glass to a user-interactive surface of the protective layer.

27. The system according to any one of claims 22 to 24 or any other claim herein wherein the protective layer comprises optically opaque material.

28. The system according to claim 27 or any other claim herein wherein outputs of the apparatus operable by the touchless interface are external to the opaque protective layer.

29. The system according to claim 28 or any other claim herein comprising light signal generator(s) (e.g. LEDs) wherein the outputs of the apparatus comprise light signals (e.g. LED light signals).

30. The system according to any one of claims 21 to 29 wherein the processor is configured to determine a user-intended pointing location on the user-facing surface of the apparatus based at least on the estimated zenith angle and estimated azimuthal angle of the conductive object.

31. The system according to claim 30 or any other claim herein wherein the processor is configured to generate a control signal for the apparatus based at least on the determined user-intended pointing location.

32. The system of any one of claims 21 to 31 comprising any of the features, combinations of features and / or sub-combinations of features of any of the preceding claims.

33. A method for estimating a user-intended pointing location on a user-facing surface of an apparatus operable with a touchless interface, the method comprising:receiving one or more capacitive sensor signals from one or more capacitive sensors, the one or more capacitive sensors sensitive to disturbances in their electric fields caused by a conductive object in proximity thereto;determining, based on the one or more capacitive sensor signals, a plurality of time-sequenced single reference center of mass points, each single reference center of mass point corresponding to a location of a center of mass of the conductive object as a single reference point detected by the one or more capacitive sensors and associated with a time stamp;generating one or more machine learning inputs based on the plurality of time-sequenced single reference center of mass points;providing the one or more machine learning inputs to a trained machinelearning algorithm comprising trainable parameters, that when executed, outputs an estimated zenith angle and an estimated azimuthal angle of the conductive object; and,determining the user-intended pointing location on the user-facing surface of the apparatus based at least on the estimated zenith angle and the estimated azimuthal angle of the conductive object,wherein the estimated zenith angle is measured from a zenith reference axis extending perpendicularly to the user-facing surface of the apparatus and the estimated azimuthal angle is measured from an azimuthal reference axis extending in parallel to the user-facing surface of the apparatus.

34. The method according to claim 33 or any other claim herein wherein the one or more machine learning inputs comprise at least one of: at least a portion of the plurality of time-sequenced single reference center of mass points; an estimated trajectory of the conductive object; average velocity value(s) of the conductive object; and, estimated acceleration value(s) of the conductive object.

35. The method according to claim 34 or any other claim herein comprising determining the estimated trajectory based on the plurality of time-sequenced single reference center of mass points.

36. The method according to claim 35 or any other claim herein wherein determining the estimated trajectory comprises, at least in part, tracing the plurality of time-sequenced single reference center of mass points in chronological order.

37. The method according to any one of claims 34 to 36 or any other claim herein comprising calculating the average velocity value between a pair of time-sequenced single reference center of mass points based on a distance between the pair of single reference center of mass points and a difference in the associated time stamps.

38. The method according to claim 37 or any other claim herein wherein each average velocity value is associated with a corresponding estimated velocity time stamp.

39. The method according to claim 38 or any other claim herein wherein the estimated velocity time stamp is determined as the temporal midpoint between the time stamps used to calculate the average velocity value.

40. The method according to any one of claims 38 to 39 or any other claim herein wherein the estimated acceleration value between a pair of average velocity values is calculated based on a difference between the pair of average velocity values and a difference between the associated estimated velocity time stamps.

41. The method according to claim 40 or any other claim herein wherein the machine learning algorithm is trained to identify stopping motions based at least on the estimated acceleration value(s), wherein the stopping motions comprise motions of the conductive object during which a velocity of the conductive object decreases to substantially zero.

42. The method according to any one of claims 33 to 41 or any other claim herein comprising continuously determining the plurality of time-sequenced single reference center of mass points within successive time windows, each time window of a predetermined and / or configurable time duration, to generate the one or more machine learning inputs associated with each corresponding time window.

43. The method according to claim 42 or any other claim herein wherein a starting time of each of the successive time windows is offset from a starting time of a preceding time window by a predetermined and / or configurable temporal offset.44 The method according to any one of claims 33 to 43 or any other claim herein comprising training the machine learning model, wherein training the machine learning model comprises: initializing values for the trainable parameters; and performing a plurality of training iterations, each training iteration comprising:(i) determining, for each motion in a set of labelled motion training data, a predicted pointing location based on current values for the trainable parameters of the machine learning model;(ii) calculating a loss value based on a comparison between the predicted pointing location and corresponding ground truth pointing location from the labelledmotion training data;(iii) computing gradients of the loss value with respect to the trainable parameters; and,(iv) updating the current values for the trainable parameters based on the computed gradients.

45. The method according to any one of claims 33 to 44 or any other claim herein wherein the motion training data for training the machine learning algorithm comprise training data generated by a robotic arm with joints and an end-effector to simulate human motions.

46. The method according to claim 45 or any other claim herein wherein the robotic arm is programmed to simulate human motions of interacting with a nearapparatus touchless interface.47 The method according to any one of claims 45 to 46 or any other claim herein wherein the robotic arm is programmed to generate varied training data by randomizing simulated trajectories of the end-effector.

48. The method according to any one of claims 45 to 46 or any other claim herein wherein the simulated human motions comprise one or more of: swiping, finger twirling, finger tapping, touching, finger hovering, finger dragging.

49. The method according to any one of claims 44 to 48 or any other claim herein wherein the motion training data comprise datasets with simulated fixed-distance motions, wherein the simulated fixed-distance motions comprises motions of the endeffector of the robotic arm stopping at or moving relative to a user-facing side of the apparatus at fixed distances to thereby emulate fixed-distance motions of a conductive object interacting with the user-facing side of the apparatus at fixed distances.

50. The method according to claim 49 or any other claim herein, wherein the machine learning algorithm is trained on datasets comprising the simulated fixed-distance motions to thereby develop the trainable parameters capable of identifyfixed-distance motions of the conductive object.

51. The method according to claim 50 or any other claim herein comprising detecting user interactions associated with the fixed-distance motions as eliciting a touch-like response from an interactive element on the user-facing side of the apparatus.

52. A system for estimating an orientation of a conductive object for interaction with an apparatus operable with a touchless interface, the system comprising:one or more capacitive sensors which are sensitive to disturbances in their electric fields caused by a conductive object in proximity thereto;a processor connected to receive one or more capacitive sensor signals from the one or more capacitive sensors, the processor configured to:determine a plurality of time-sequenced single reference center of mass points based on the one or more capacitive sensor signals, each single reference center of mass point corresponding to a location of a center of mass of the conductive object as a single reference point and a corresponding time stamp;generate one or more machine learning inputs based on the plurality of time-sequenced single reference center of mass points;execute a trained machine learning algorithm with the one or more machine learning inputs to infer an estimated zenith angle and an estimated azimuthal angle of the conductive object; and,determine the user-intended pointing location on the user-facing surface of the apparatus based at least on the estimated zenith angle and the estimated azimuthal angle of the conductive object,wherein the estimated zenith angle is measured from a zenith reference axis extending perpendicularly to the user-facing surface of the apparatus and the estimated azimuthal angle is measured from an azimuthal reference axis extending in parallel to the user-facing surface of the apparatus.

53. The system according to claim 52 or any other claim herein wherein the apparatus comprises a protective layer overlaid on the capacitive sensors.

54. The system according to claim 53 or any other claim herein wherein the protective layer is spaced apart from a user-facing side of the apparatus by a separation distance to define an air gap between the protective layer and the userfacing side of the apparatus to thereby protect the user-facing side of the apparatus from physical damage due to deformation of the protective layer (e.g. when an external force is applied against the protective layer) and / or to provide heat insulation to the user-facing side of the apparatus from ambient heat.

55. The system according to claim 53 or 54 or any other claim herein wherein motion training data for the machine learning algorithm comprise datasets with simulated motions emulating a conductive object coming into contact with the protective layer when interacting with the touchless interface to thereby train the machine learning algorithm to identify motion patterns specific to conductive object coming into contact with the protective layer.

56. The system according to any one of claims 53 to 55 or any other claim herein wherein the protective layer comprises optically transparent material.

57. The system according to claim 56 or any other claim herein wherein the protective layer is made of fiber optic glass and the apparatus comprises a display configured to display images wherein the displayed images are optically transmitted by the fiber optic glass to a user-interactive surface of the protective layer.

58. The system according to any one of claims 53 to 55 or any other claim herein wherein the protective layer comprises optically opaque material.

59. The system according to claim 58 or any other claim herein wherein outputs of the apparatus operable by the touchless interface are external to the opaque protective layer.

60. The system according to claim 59 or any other claim herein comprising light signal generator(s) (e.g. LEDs) wherein the outputs of the apparatus comprise light signals (e.g. LED light signals).

61. The system according to any one of claims 52 to 60 wherein the processor is configured to determine a user-intended pointing location on the user-facing surface of the apparatus based at least on the estimated zenith angle and estimated azimuthal angle of the conductive object.

62. The system according to claim 61 or any other claim herein wherein the processor is configured to generate a control signal for the apparatus based at least on the determined user-intended pointing location.

63. The system of any one of claims 52 to 62 comprising any of the features, combinations of features and / or sub-combinations of features of any of the preceding claims.

64. Apparatus having any new and inventive feature, combination of features, or sub-combination of features as described herein.

65. Methods having any new and inventive steps, acts, combination of steps and / or acts or sub-combination of steps and / or acts as described herein.

Images