Motion-based controllers

Abstract

An interactive installation that enables physical activity to be used as a control mechanism for entertainment experiences is disclosed. The installation includes a physical engagement apparatus, motion sensors to detect engagement of a user with the physical engagement apparatus and to generate a corresponding motion signal, and an electronic device configured to render an audio arrangement. Rendering of the audio arrangement is modulated by an input signal generated based at least in part on the motion signal. The system is adaptable to a wide range of physical engagement apparatus, including exercise and play equipment such as swings and stationary bicycles, without requiring structural modification. Motion sensing can be performed using sensors mounted on the apparatus or sensors integrated into a handheld device of the user. User movements such as swinging, pedalling, leaning or shifting posture can be translated into engagement metrics used to dynamically modulate audio or gaming parameters.

Description

MOTION-BASED CONTROLLERSCROSS-REFERENCE TO RELATED APPLICATIONS

[0000] This application claims the benefit of, and priority to, United States Provisional Patent Applications No. 63 / 749,063, filed January 24, 2025, and entitled “ANGULAR MOTION-BASED CONTROLLER”, and No. 63 / 796,020, filed April 29, 2025, and entitled “TILT-BASED CONTROLLER AND ACTIVITY-BASED METACONTROLLER”, the disclosures of which are hereby incorporated by reference in their entirety.TECHNICAL FIELD

[0001] The technical field relates to input arrangements for interaction between user and computer, and more specifically to systems and methods for controlling an entertainment application based on motion or position of a user.BACKGROUND

[0002] Interactive entertainment systems can rely on natural user movement to control digital content, offering more intuitive and engaging experiences than traditional handheld controllers. This trend has led to the integration of sensors, motion-tracking devices and computational models into a wide range of physical environments, from public art installations to video game systems and personal exercise equipment. By mapping human physical activity to virtual actions, creators can offer novel immersive interactions that motivate users and reduce the need for complex or specialized hardware.

[0003] Despite these advances, existing motion-based control systems suffer from significant limitations. Many systems rely on sensors that are permanently installed within a physical apparatus, such as sensors embedded in a swing frame or in exercise equipment. Such installations restrict portability and limit the ability to use the control system in multi-station environments or shared public spaces, or with off-the-shelf exercise machines. Moreover, some systems focus solely onmeasuring simple forms of linear motion and do not adequately capture complex or angular movements that would enable richer and more expressive control in virtual environments.

[0004] As an example, existing interactive exercise systems that link stationary bicycles to video games typically require proprietary equipment, specialized tilting mechanisms and / or modified handlebars to enable lateral and / or directional input. These approaches either constrain compatibility or require significant mechanical adaptation. As another example, existing sensor-based systems for swings are typically used for safety or maintenance purposes rather than for entertainment control. They typically embed sensors within the swing’s posts, hinges or support elements, preventing use across multiple swings or in settings where hardware modification is not practical. For instance, as shown in figure 1A, Korean Patent No. 101937988 as well as Korean Publication No. 20180134545 use at least one sensor 1122 permanently mounted on at least one corresponding post 112 of a swing apparatus 100, Indian Publication No. 202111002538 as well as US Patent No. 7,905,791 use a sensor 162 permanently mounted at the hinge 160, and US Patent No. 9,868,071 uses a sensor 1422 permanently mounted at the support element 142. A sensor 1122 can be configured to detect the swing support element’s traversal past the post 112, with measurements reflecting the swing’s position relative to the post. Such sensors are typically used to estimate the swing’s velocity, acceleration, and angular displacement by analyzing the timing and frequency of the support element’s passes. Sensors 1422 or 162, on the other hand, are positioned in locations where they can directly detect changes in the angular position of the support element 142, particularly as the support element moves along its swinging trajectory.SUMMARY

[0005] There is a need for an intuitive and widely compatible solution that can measure a user’s motion or pose for use as an entertainment system or application controller while remaining adaptable to generic underlying physical apparatus.There is also a need for systems capable of interpreting physical movements, such as swinging, pedalling and / or leaning, into meaningful control signals for games or entertainment applications, while remaining widely compatible, portable and adaptable to diverse environments.

[0006] The present disclosure provides a system and method for using various exercise or play equipment, such as a stationary bicycle or a swing, as a controller. As an example, a controller configured to be used with a stationary bicycle can determine lateral movement based on the user’s pose rather than requiring modifications to the bicycle itself. By leveraging sensors to track shifts in body position, such a controller can allow users to steer an avatar or navigate a virtual environment in a natural and responsive manner. This approach eliminates the need for specialized handlebars, resistance-based input mechanisms and / or proprietary exercise equipment, making it accessible for use with a wide range of off-the-shelf stationary bicycles. As another example, a controller configured to be used with a swing can detect the angular motion of a handheld device manipulated by the user as they swing back and forth, thereby deriving a momentum-based input signal that reflects the amplitude or energy of the swinging motion to influence an entertainment or virtual experience. Because the motion is captured using a portable device rather than sensors permanently installed on the swing structure, the system can be deployed across multiple swings without requiring modification of the equipment.

[0007] Different aspects are provided by way of examples.

[0008] In an aspect, an interactive installation is provided. The installation includes a physical engagement apparatus configured to support physical activity by a user; at least one motion sensor configured to detect an engagement of the user with the physical engagement apparatus and to generate at least one motion signal indicative of the engagement; and an electronic device configured to render an audio arrangement, the rendering being modulated by an input signal generated based at least in part on the motion signal.

[0009] In some embodiments, the motion signal is based on a level of kinetic energy generated by the engagement of the user with the physical engagement level.

[0010] In some embodiments, the input signal is generated based on a momentum level computed by aggregating succesive motion signals generated by the motion sensor over a sliding time window.

[0011] In some embodiments, the physical engagement apparatus is a swing apparatus, and wherein the momentum level is computed based on at least one of: a magnitude of angular displacement over the sliding time window, a rate of rotation around at least one axis, a level of change in momentum between a present time and a previous time, and changes in rotation direction.

[0012] In some embodiments, the physical engagement apparatus is a stationary bicycle, and wherein the momentum level is computed based on at least one of: a rotation rate of a crankset and / or pedal assembly, a cadence of user pedalling over the sliding time window, a magnitude of angular displacement of the crankset over the sliding time window, and a level of change in rotation speed between a present time and a previous time.

[0013] In some embodiments, the installation further includes at least one additional physical engagement apparatus configured to support additional physical activity by at least one additional user; and at least one additional sensor configured to detect an additional engagement of the additional user with the additional physical engagement apparatus and to generate at least one additional motion signal indicative thereof, wherein the input signal is generated based at least in part on the additional motion signal.

[0014] In some embodiments, the audio arrangement comprises a plurality of tracks, and wherein a rendering of each track is modulated by an input signal generated based at least in part on the motion signal associated with the user orone of the additional motion signal associated with one or the at least one additional user.

[0015] In some embodiments, the motion sensor includes at least one apparatusmounted sensor secured to the physical engagement apparatus.

[0016] In some embodiments, the apparatus-mounted sensor includes an infrared sensor configured to detect the engagement of the user with the physical engagement apparatus.

[0017] In some embodiments, the apparatus-mounted sensor includes an imaging sensor configured to acquire images of the user and infer from the images the engagement of the user with the physical engagement apparatus.

[0018] In some embodiments, the motion sensor includes at least one devicebased sensor installed in a handheld device of the user configured to detect movements of the user and infer from the movements the engagement of the user with the physical engagement apparatus.

[0019] In some embodiments, the electronic device is the handheld device.

[0020] In some embodiments, the device-based sensor comprises at least one of: an inertial measurement unit, a gyroscope, an accelerometer, and a magnetometer.

[0021] In some embodiments, the installation further includes a computer-readable code displayed on or near the physical engagement apparatus, wherein the electronic device is configured to read the computer-readable code and obtain the audio arrangement based at least in part on the computer-readable code.

[0022] In some embodiments, the electronic device further comprise at least one geolocation sensor configured to determine a geographic location, and the electronic device is configured to obtain the audio arrangement based at least in part on the geographic location.

[0023] In another aspect, a method of operating an interactive installation is provided. The method includes detecting an engagement of a user with a physical engagement apparatus; generating at least one motion signal indicative of the engagement; and rendering an audio arrangement, the rendering being modulated by an input signal generated based at least in part on the motion signal.

[0024] In some embodiments, the motion signal is based on a level of kinetic energy generated by the engagement of the user with the physical engagement level.

[0025] In some embodiments, the input signal is generated based on a momentum level, further comprising computing the momentum level by aggregating succesive motion signals generated over a sliding time window.

[0026] In some embodiments, the physical engagement apparatus is a swing apparatus, and wherein the momentum level is computed based on at least one of: a magnitude of angular displacement over the sliding time window, a rate of rotation around at least one axis, a level of change in momentum between a present time and a previous time, and changes in rotation direction.

[0027] In some embodiments, the physical engagement apparatus is a stationary bicycle, and wherein the momentum level is computed based on at least one of: a rotation rate of a crankset and / or pedal assembly, a cadence of user pedalling over the sliding time window, a magnitude of angular displacement of the crankset over the sliding time window, and a level of change in rotation speed between a present time and a previous time.

[0028] In some embodiments, the method further includes detecting at least one additional engagement of at least one additional user with at least one additional physical engagement apparatus; and generating at least one additional motion signal indicative of the additional engagement, wherein the input signal is generated based at least in part on the additional motion signal.

[0029] In some embodiments, the audio arrangement comprises a plurality of tracks, and wherein rendering the audio arrangement comprises rendering each track, the rendering of each track being modulated by an input signal generated based at least in part on the motion signal associated with the user or one of the additional motion signal associated with one or the at least one additional user.

[0030] In some embodiments, detecting the engagement and / or generating the motion signal is at least in part performed by at least one apparatus-mounted sensor secured to the physical engagement apparatus.

[0031] In some embodiments, the apparatus-mounted sensor comprises an infrared sensor configured to detect the engagement of the user with the physical engagement apparatus.

[0032] In some embodiments, the apparatus-mounted sensor comprises an imaging sensor configured to acquire images of the user and infer from the images the engagement of the user with the physical engagement apparatus.

[0033] In some embodiments, detecting the engagement and / or generating the motion signal is at least in part performed by at least one device-based sensor installed in a handheld device of the user configured to detect movements of the user and infer from the movements the engagement of the user with the physical engagement apparatus.

[0034] In some embodiments, the device-based sensor comprises at least one of: an inertial measurement unit, a gyroscope, an accelerometer, and a magnetometer.

[0035] In some embodiments, the method further includes reading by an electronic device the computer-readable code and obtaining by the electronic device the audio arrangement based at least in part on the computer-readable code.

[0036] In some embodiments, the method further includes determining by an electronic device a geographic location and obtaining by the electronic device the audio arrangement based at least in part on the geographic location.

[0037] In a further aspect, an angular motion-based controller implemented in a handheld device is provided. The controller includes at least one motion sensor configured to measure a magnitude of an angular movement generated by a user engaging with a physical engagement apparatus; a processor configured to compute a momentum level of the handheld device based at least in part on the magnitude of the angular movement, a memory comprising instructions corresponding to at least one application, the application being configured to use the momentum level as an input parameter.

[0038] In some embodiments, the angular movement is causes by an individual holding the handheld device while using a swing.

[0039] In some embodiments, the application is an audio playback application configured to render an audio recording, the rendering being modulated on the input parameter.

[0040] In some embodiments, the processor is configured to receive at least one additional momentum level of at least one additional handheld device from at least one additional controller, wherein the at least one additional momentum level of the at least one additional handheld device is provided as an additional input parameter to the application.

[0041] In some embodiments, the audio recording comprises a plurality of tracks, wherein a rendeing of each one of the tracks is modulated based at least on the input parameter associated with the handheld device or the additional input parameter associated with the additional handheld device.

[0042] In yet another aspect, a method of operating an angular motion-based controller is provided. The method includes measuring a magnitude of an angular movement generated by a user engaging with a physical engagement apparatus;computing a momentum level of the handheld device based at least in part on the magnitude of the angular movement, providing the momentum level as an input parameter of at least one application.

[0043] In some embodiments, the angular movement is caused by an individual holding the handheld device while using a swing.

[0044] In some embodiments, the method further includes rendering an audio recording by the application, the rendering being modulated on the input parameter.

[0045] In some embodiments, the method further includes computing at least one additional momentum level associated with at least one additional handheld device from at least one additional controller; and providing the at least one additional momentum level of the at least one additional handheld device as an additional input parameter to the application.

[0046] In some embodiments, the audio recording comprises a plurality of tracks, further comprising rendeing of each one of the tracks, the rendeing of each one of the tracks being modulated based at least on the input parameter associated with the handheld device or the additional input parameter associated with the additional handheld device.

[0047] In yet a further aspect, a metacontroller system is provided. The system includes an entertainment device configured to render a virtual environment based at least in part on control signals; a controller comprising a plurality of controls and configured to generate the control signals based on user interactions with the controls, the controls comprising a directional control and an action control; a physical engagement apparatus comprising at least one motion sensor configured to generate a motion signal based on detecting movement indicative of an engagement of the user with the physical engagement apparatus; and a metacontroller configured to receive the control signals from the controller and the motion signal from the exercise apparatus, and to forward at least a portion of thecontrol signals to the electronic entertainment device based on the movement signal.

[0048] In some embodiments, the metacontroller is configured to forward the control signals only when the movement signal indicates detected movement.

[0049] In some embodiments, the metacontroller is configured to forward a first portion of the control signals corresponding to the action control at all times, and to forward a second portion of the control signals corresponding to the directional control only when the movement signal indicates detected movement.

[0050] In some embodiments, the metacontroller is configured to store a duration of the movement signal indicating detected movement and to forward the control signals for an interval of time based on the duration.

[0051] In yet another aspect, a method for controlling an electronic entertainment device based on a controller and a physical engagement apparatus is provided. The method includes generating control signals based on actuation of the controller; detecting motion indicative of engagement of a user with the physical engagement apparatus; forwarding at least a portion of the control signals to the electronic entertainment device based on the motion detection; and rendering by the electronic entertainment device a virtual environment based on the forwarded portion of the control signals.

[0052] In some embodiments, the forwarding comprises forwarding the control signals only when the motion is detected.

[0053] In some embodiments, the forwarding comprises: forwarding a first portion of the control signals corresponding to the action control at all times; and forwarding a second portion of the control signals corresponding to the directional control only when the movement signal indicates detected movement.

[0054] In some embodiments, the method further includes storing a duration of the motion being detected, wherein forwarding comprises forwarding the control signals for an interval of time based on the duration.

[0055] In yet a further aspect, a metacontroller is provided. The metacontroller is configured to receive control signals from a controller comprising a plurality of controls and configured to generate the control signals based on user interactions with the controls, the controls comprising a directional control and an action control, and motion signal from a physical engagement apparatus comprising at least one sensor configured to generate the motion signal based on detecting engagement of a user with the physical engagement apparatus; and forward at least a portion of the control signals to an entertainment device configured to render a virtual environment based at least in part on control signals based on the movement signal.

[0056] In yet another aspect, a method of operating a metacontroller is provided. The method includes receiving control signals from a controller comprising a plurality of controls and configured to generate the control signals based on user interactions with the controls, the controls comprising a directional control and an action control, and motion signal from a physical engagement apparatus comprising at least one sensor configured to generate the motion signal based on detecting engagement of a user with the physical engagement apparatus; and forwarding at least a portion of the control signals to an entertainment device configured to render a virtual environment based at least in part on control signals based on the movement signal.

[0057] In yet a further aspect, an interactive installation is provided. The installation includes a display; an electronic entertainment device configured to render an avatar in a virtual environment to the display based on first and second control signals; a stationary bicycle comprising a crankset and a motion sensor operably coupled to the electronic entertainment device and configured to function as a first controller of the electronic entertainment device by generating the firstcontrol signals in response to a user actuating the crankset; a tilt-based controller operably coupled to the electronic entertainment device and configured to function as a second controller of the electronic entertainment device, the tilt-based controller comprising: an imaging sensor configured to acquire an image of the user, a memory comprising a machine learning model trained to accept the image as input and predict coordinates of at least two primary anatomical landmarks, and a processor configured to: generate the coordinates of the at least two primary anatomical landmarks by executing the machine learning model, generate the coordinates of two secondary anatomical landmarks based on the at least two primary anatomical landmarks, compute an angle formed by a straight line connecting the two secondary anatomical landmarks and a reference axis, and generate the second control signal based on the angle.

[0058] In yet another aspect, an electronic entertainment system is provided. The system includes a display; an electronic entertainment device configured to render an avatar in a virtual environment to the display based on control signals; a controller comprising a plurality of controls and configured to generate the control signals based on user interactions with the controls, the controls comprising a directional control and an action control; a stationary bicycle comprising a crankset and a motion sensor configured to generate a movement signal in response to a user actuating the crankset; and a metacontroller operably coupled to the electronic entertainment device, the controller and the motion sensor, the metacontroller comprising a processor configured to: receive the control signals from the controller, forward the control signals associated with the action control to the electronic entertainment device, and in response to receiving the movement signal, forward the control signals associated with the directional control to the electronic entertainment device.

[0059] In yet a further aspect, a tilt-based controller system is provided. The system includes an exercise apparatus; a display; an electronic entertainment device configured to render to the display a virtual environment based at least in part on control signals; an imaging sensor configured to acquire an image of anindividual using the exercise apparatus; and a processor configured to: run a machine learning model trained to accept the image as input and predict coordinates of at least two primary anatomical landmarks, and compute an angle formed by: a straight line connecting two secondary anatomical landmarks based on the coordinates of the at least two primary anatomical landmarks, and a reference axis, wherein the angle is provided as part of the control signals to the electronic entertainment device.

[0060] In some embodiments, the angle is the result of the individual leaning to a side.

[0061] In some embodiments, the virtual environment comprises an avatar and wherein the entertainment device is configured to render a lateral movement of the avatar based on the angle.

[0062] In yet another aspect, a method for controlling an electronic entertainment device based on a pose of an individual is provided. The method includes capturing an image of the individual; predicting coordinates of at least two anatomical landmarks; computing the pose of the individual based on the coordinates of the at least two primary anatomical landmarks; and rendering by the electronic entertainment device a virtual environment based on the pose of the individual.

[0063] In some embodiments, the pose indicates whether the individual is leaning to a side.

[0064] In yet a further aspect, a tilt-based controller is provided. The controller includes an imaging sensor configured to acquire an image of an individual using an exercise apparatus; and a processor configured to: run a machine learning model trained to accept the image as input and predict coordinates of at least two primary anatomical landmarks, and compute an angle formed by: a straight line connecting two secondary anatomical landmarks based on the coordinates of the at least two primary anatomical landmarks, and a reference axis, wherein the angle is provided as part of the control signals to an electronic entertainment deviceconfigured to render to a display a virtual environment based at least in part on control signals.BRIEF DESCRIPTION OF THE DRAWINGS

[0065] For a better understanding of the embodiments described herein and to show more clearly how they may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings which show at least one exemplary embodiment.

[0066] Figure 1A to 1C are illustrations of swing apparatus in accordance with different embodiments.

[0067] Figure 2A is an illustration of a handheld device usable as a controller in accordance with an embodiment.

[0068] Figures 2B and 2C are schematics of systems for rendering an audio arrangement associated with a swing apparatus in accordance with two embodiments.

[0069] Figure 3 is a flowchart of a method for rendering an audio arrangement associated with a swing apparatus in accordance with an embodiment.

[0070] Figure 4 is a schematic of a generic stationary bicycle, in accordance with an embodiment.

[0071] Figure 5A is a schematic of a tilt-based controller system, in accordance with an embodiment.

[0072] Figures 5B and 5C are illustrations of anatomical landmarks and tilt computation, in accordance with an embodiment.

[0073] Figure 6 is a flowchart of a method for operating a tilt-based controller, in accordance with an embodiment.

[0074] Figure 7 is a schematic of a metacontroller system, in accordance with an embodiment.

[0075] Figure 8 is a flowchart of a method for operating a metacontroller, in accordance with an embodiment.DETAILED DESCRIPTION

[0076] It will be appreciated that, for simplicity and clarity of illustration, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements or steps. In addition, numerous specific details are set forth in order to provide a thorough understanding of the exemplary embodiments described herein. However, it will be understood by those of ordinary skill in the art that the embodiments described herein may be practised without these specific details. In other instances, well-known methods, procedures and components have not been described in detail so as not to obscure the embodiments described herein. Furthermore, this description is not to be considered as limiting the scope of the embodiments described herein in any way but rather as merely describing the implementation of the various embodiments described herein.

[0077] As used herein, the term “physical engagement apparatus” refers generally to any apparatus, structure, device and / or installation configured to enable a user to engage in physical activity, bodily movement and / or exertion through interaction therewith, whether for recreational, leisure, artistic, educational, therapeutic and / or fitness purposes. A physical engagement apparatus can be configured to support, guide, constrain and / or respond to user motion and can be stationary or movable, passive or active, mechanical, electromechanical and / or electronically augmented. By way of non-limiting examples, a physical engagement apparatus can include a swing apparatus, a stationary bicycle, a rowing machine, an elliptical trainer, a treadmill, a balance or wobble platform, a climbing or hanging structure, a rotating or oscillating device, a pedal-based or crank-based apparatus, a spring-based apparatus, a playground installation, aninteractive sculpture, an outdoor fitness installation and / or other motion-based equipment enabling physical engagement by one or more users.

[0078] In the present disclosure, swing apparatus and stationary bicycles are particularly described in certain embodiments as illustrative examples of a physical engagement apparatus. It can be appreciated, however, that the systems, methods and techniques described herein are not limited to these examples and can be applied to other types of physical engagement apparatus, in some case with required adaptations that will be readily apparent to those skilled in the art. Accordingly, where reference is made in the disclosure below to a particular type of physical engagement apparatus, such as a swing apparatus or a stationary bicycle, the corresponding described features can likewise apply to other types of physical engagement apparatus capable of enabling user motion and / or physical exertion, with appropriate modifications that will be apparent to those skilled in the art.

[0079] As used herein, the term “motion sensor” refers generally to any sensor, sensing system and / or combination of sensors configured to detect, measure, infer and / or estimate “physical engagement” of a user with a physical engagement apparatus and to generate one or more signals indicative thereof. A motion sensor can be configured to directly sense motion, position, displacement, velocity, acceleration, rotation and / or force, and / or to indirectly infer user activity, engagement level and / or exertion based on sensed events, periodicity, intensity and / or temporal patterns. A motion sensor can comprise one or more optical sensors, infrared sensors, proximity sensors, inertial sensors, magnetic sensors, pressure sensors, force sensors, strain gauges, encoders, switches and / or other sensing elements, whether used alone or in combination, and can be implemented as discrete components, integrated sensor modules and / or part of a larger sensing subsystem.

[0080] By way of non-limiting examples, a motion sensor can be configured to detect passages of a user or user body part, such as a foot, hand and / or torso,detect rotations or oscillations of a mechanical element such as a crankset, pedal, swing platform or suspension element, detect periodic motion associated with repetitive physical activity, and / or generate a control signal indicative of user actuation, momentum, cadence, intensity and / or engagement level. In the present disclosure, a motion sensor is described in different embodiments as a sensor configured to generate a control signal in response to the user engaging with a specific type of physical engagement apparatus. However, it will be appreciated that the same concepts apply to other types of motion sensors and other types of physical engagement apparatus, with necessary adaptations. Accordingly, where reference is made in the disclosure below to a particular type of motion sensor or sensing configuration, the corresponding described features can likewise apply to different sensing modalities and arrangements capable of sensing physical engagement, with appropriate modifications that will be apparent to those skilled in the art.

[0081] In the present disclosure, signals generated by a sensor can be referred to generally as “motion signals”, which can include raw sensor outputs, processed signals and / or derived metrics indicative of physical engagement of a user with a physical engagement apparatus. Such motion signals can be used directly or indirectly as input signals to one or more systems, including without limitation entertainment systems, media rendering systems, interactive installations and / or other responsive systems. In some embodiments, a motion signal can act directly or indirectly as a primary control input, whereby characteristics of the physical engagement directly influence operation of the controlled system. In other embodiments, the motion signal can act as a secondary or meta-control input, for example by directly or indirectly modulating, scaling, enabling, weighting and / or otherwise influencing input received from a separate controller, such as a handheld device, a user interface, a game controller and / or a generic input device.

[0082] In some embodiments, one or more values derived from motion signals can be expressed as abstracted engagement metrics, such as a momentum level, a kinetic energy level, an intensity level and / or other normalized indicators of useractivity. Such metrics can be computed based on one or more sensed parameters including, without limitation, detected events, frequency, cadence, velocity, acceleration, amplitude, periodicity and / or temporal patterns of motion. A momentum level or kinetic energy level does not necessarily correspond to physical momentum or kinetic energy in a strict mechanical sense, but can instead represent a relative, normalized and / or application-specific measure of engagement suitable for controlling system behaviour. Accordingly, references herein to momentum level, kinetic energy level and similar terms are intended to encompass both physically grounded measurements and higher-level derived or inferred values used for control, adaptation and / or interaction purposes.

[0083] With reference to figures 1B and 1C, exemplary physical engagement apparatus corresponding to swing apparatus 100 are shown.

[0084] Each swing apparatus 100 includes one or more swing(s) 140 each adapted to support at least one user 50 moving back and forth on the swing.

[0085] Each swing apparatus 100 can include a frame 110, including at least two posts 112 and one crossbar 114. The one or more swings 140 each includes a support element 142 and at least one suspension element 144. Each suspension element 144 can be attached to the crossbar 114 via a hinge 160 adapted to allow movement of the support element 142 along at least one axis.

[0086] A swing apparatus 100, as described herein, refers to any mechanical structure designed to support and enable the motion of one or more swings for recreational or functional purposes. Such swing apparatus may include, but is not limited to, swing sets found in public parks, playgrounds, gardens, amusement parks, schoolyards, or private residential areas. It may also encompass swing chairs, porch swings, garden swings, or other similar devices designed for human use or other types of suspended seating. The apparatus 100 can be used for various activities including play, relaxation, exercise and / or entertainment. Swing apparatus 100 can also be designed for specific uses, such as those for children, adults or even animals, and may include one or more swings 140.

[0087] As mentioned above, each swing apparatus 100 can include a frame 110, which serves as the structural foundation of the apparatus. The frame 110 is typically formed by multiple interconnected components that provide stability and support for the swing(s) 140. At a minimum, the frame 110 includes two posts 112 and one crossbar 114. These posts 112 are generally upright vertical supports that anchor the swing apparatus to the ground or other stable foundation. The posts 112 may be made of a variety of materials, such as metal, e.g., steel and / or aluminum, wood, e.g., oak, pine and / or treated lumber, plastic, and / or composite materials, with the choice of material affecting the strength, durability and aesthetic appearance of the frame.

[0088] Each post 112 may have a variety of configurations, including but not limited to straight posts, tapered posts, or arched posts. The posts may have cross-sectional shapes that include round, square, rectangular and / or hexagonal profiles. These posts 112 may also be fixed into the ground through various methods, including direct embedding in concrete, installation in ground sleeves, bolted to a base plate and / or through surface-mounting systems for modular or movable swing apparatuses. The height of the posts 112 can vary, depending on the intended use of the swing, with heights ranging from relatively low posts, e.g., for children, to higher posts, e.g., for adult swing sets or swing chairs.

[0089] The crossbar 114, which extends horizontally between posts 112, can act as the anchor point for the one or more swings 140. The crossbar 114 can provide structural support to the entire swing apparatus and can be made of materials similar to the posts 112, including metal, wood, or composite materials. The crossbar 114 may for instance have a circular, square and / or rectangular crosssection, and may be designed to bear the weight and stress generated by the swinging motion. In addition to its function as a load-bearing component, the crossbar 114 can also serve an aesthetic role, with ornamental finishes, coatings and / or designs. It can be suspended between the posts 112 by various fastening systems such as bolts, welding and / or integral construction.

[0090] Each swing 140 within the swing apparatus 100 includes at least one support element 142, which serves as the seat or surface for the person or object intended to be suspended and swung. The support element 142 may come in many different forms and materials depending on the intended use. For example, in a playground swing set, the support element may be a flat or contoured seat made from plastic, rubber, wood and / or metal, and it may optionally include padding or other comfort-enhancing features. In other swing apparatus 100, such as those used for relaxation, the support element may be a chair-like structure made of cushioned fabric, wicker and / or other comfortable materials.

[0091] The support element 142 can also be designed to accommodate various types of users, with adjustable components for children, adults or even pets. For example, some swing support elements 142 are designed to be reclining or adjustable in angle to provide a more comfortable or relaxing experience. Other swing support elements 142 may feature additional safety features such as side rails, footrests and / or built-in harnesses for added security. The size, shape and material of the support element 142 may vary widely based on the type of swing apparatus and its intended application.

[0092] Each support element 142 is suspended by at least one suspension element 144, which is typically a flexible or resilient cord, rope, chain or other tethering mechanism that connects the support element to an attachment point or structure such as the crossbar 114. The suspension element 144 is responsible for supporting the weight of the support element 142 and the user 50 while allowing movement along at least one axis, such as a pivot axis defined by the crossbar, extending perpendicular to the length of the crossbar and parallel to the ground, around which the swing support element pivots, but sometimes along multiple axes for more dynamic swinging motion.

[0093] The suspension element 144 can be made from various materials such as nylon, polypropylene, polyester, steel cable and / or other materials designed to withstand tension and wear. The material used for the suspension element 144can have sufficient strength, flexibility and durability to support the weight of the user 50 and withstand environmental elements such as sun, rain and / or temperature fluctuations. The suspension element 144 can also include protective coverings or coatings, such as rubber or vinyl, to prevent wear and tear, increase comfort and / or improve safety by reducing friction.

[0094] In certain swing designs, the suspension element 144 may be reinforced with additional features, such as braided or twisted fibres, additional support cords, or padding. In some embodiments, the suspension element 144 may be adjustable in length to modify the height or position of the support element 142, and / or may be equipped with locking mechanisms to prevent unwanted adjustments. Additionally or alternatively, a combination of suspension elements or multiple suspension points may be used for added stability or comfort.

[0095] Each suspension element 144 is typically attached to a crossbar 114 via a hinge 160, which allows the support element 142 to move freely during use. The hinge 160 serves as the pivotal point of attachment and permits the swinging motion of the support element 142 along at least one axis, typically the vertical axis, allowing the support element to move back and forth in a controlled, oscillating motion. The hinge 160 may be a simple pin-and-bore hinge or a more complex mechanism, such as a swivel or rotating joint, which accommodates both vertical and horizontal movement. The hinge 160 can be made from various materials, including steel, brass, aluminum and / or plastic, and may be designed to be corrosion-resistant or weatherproof to withstand outdoor conditions.

[0096] The hinge 160 may also be designed to include additional features, such as bushings, bearings and / or lubricants, to reduce friction and wear on the hinge mechanism during use. The hinge 160 may be configured to allow the support element 142 to swing freely along a single axis, or it may support multidimensional movement to create a more dynamic swinging motion. Some swing apparatus may feature a combination of hinges and rotating elements to allow for a more complex range of motion.

[0097] A swing apparatus 100 can include components such as sensors configured to acquire data that may be used to compute an angular motion of a support element 142. These sensors can be configured to measure motion in terms of angular and / or linear velocity, e.g., in rad / s or m / s, and / or acceleration, e.g., in rad / s2or m / s2. Additionally, the sensors may provide data to compute a relative motion index, a kinetic or dynamic energy index, or a combination of these parameters, all of which are related to the swing’s motion along one or more pivot and / or tangential axes.

[0098] As mentioned above, in typical prior art embodiments, sensors are permanently embedded in the swing apparatus 100, which can introduce several potential drawbacks. One concern is the durability of the embedded sensors. Given the dynamic nature of the swinging motion, the sensors are subjected to continuous physical stress, particularly at attachment points such as hinges or suspension points. Over time, this repetitive motion can lead to wear and tear, causing degradation in sensor accuracy or eventual failure. Additionally, exposure to environmental elements, such as rain, snow, and ultraviolet (UV) radiation, can negatively affect the performance of the sensors if the swing apparatus is used outdoors. Moisture infiltration or UV exposure may lead to corrosion, sensor malfunction, ora reduction in the longevity of the embedded components, thereby increasing the need for maintenance or replacement.

[0099] Another drawback of embedding sensors in the swing apparatus 100 is the increased manufacturing complexity and cost. Integrating sensors into the swing set requires specialized hardware, precise installation, and calibration. This adds to the production cost, making the swing apparatus 100 more expensive to manufacture and potentially less accessible to consumers. The need for careful assembly to ensure proper alignment and functionality also increases the complexity of the manufacturing process. Any misalignment or installation defects may lead to incorrect sensor readings, adversely affecting the accuracy of motion data and potentially compromising the performance of any safety systems or motion algorithms that rely on this data.

[0100] Moreover, embedded sensors may suffer from calibration and alignment challenges. Since these sensors are fixed in position within the swing apparatus 100, any misalignment, even minor, can result in significant measurement errors. Such errors could impact the precision of data related to angular motion, velocity or acceleration, and ultimately affect the reliability of the entire motion detection system. The need for periodic recalibration could introduce additional maintenance requirements. Furthermore, once the sensors are embedded, they are typically not adjustable, which limits the flexibility to recalibrate or reposition them to correct any issues with accuracy or alignment.

[0101] Finally, sensor failure or inaccuracies in sensor systems embedded in the swing apparatus 100 can lead to difficulty in diagnosing and repairing problems. When sensors are embedded within structural components of the swing, they are not easily accessible for troubleshooting or replacement. This could result in extended downtime for the swing apparatus 100, as technicians would need to disassemble portions of the structure to repair or replace malfunctioning sensors, adding to the overall operational cost and reducing the system’s reliability.

[0102] The present disclosure circumvents the aforementioned downsides by allowing the swing user to provide their own sensor or set of sensors, integrated within a general-purpose handheld device 200. This approach eliminates the need for permanently embedded sensors, thereby avoiding issues related to sensor durability, calibration and / or maintenance. By leveraging a user-supplied device 200, the system provides greater flexibility, reduces manufacturing complexity and cost, and offers easy access for troubleshooting or upgrading the sensor as needed.

[0103] With reference to figures 2A and 2B, an exemplary handheld device 200 is shown. The device 200 can be used to implement a controller that allows users to control an application, e.g., an entertainment application, based on angular motion. As an example, the device 200 could be held by a user moving back and forth on a swing, e.g., in the hand, in a pocket or in a bag, and used as a controllerto control media playback in the application based on an amplitude or speed of detected angular motion. Advantageously, this provides for a system that is capable of measuring the motion of a swing, or an angular motion akin to that of a swing, while not being embedded or permanently affixed to an apparatus. As explained below, this approach is adaptable to measure the motion of a plurality of swings, for instance swings in a multi-swing set, to control a single shared experience.

[0104] A handheld device 200, as described herein, can refer to any compact, portable device capable of being conveniently held and operated by an individual, typically within one hand or two hands, and designed for a variety of tasks including, but not limited to, communication, computation, navigation, media consumption, gaming and / or Internet browsing. Examples of such handheld devices include, but are not limited to, smartphones, tablet computers, personal digital assistants (PDAs), handheld gaming consoles, portable media players, smartwatches, e-readers and other portable computing devices designed for personal use. The devices may come in a range of form factors, including, but not limited to, slab-like devices, foldable or flip devices, and devices with detachable components or screens.

[0105] The handheld device 200 can include at least one processor 210, which may be configured to execute a set of instructions stored in the memory 220 of the device 200. The processor 210 can be any type of microprocessor or microcontroller capable of carrying out instructions to perform various functions associated with the device’s operations. The processor may be a single core processor, a multi-core processor or a hybrid processor. Examples of processors suitable for device 200 include, but are not limited to, ARMTM-based processors such as the ARM Cortex™ series, Intel™ processors such as the Atom™ or Core™ series, Qualcomm™ Snapdragon™ processors, Apple™ A-series™ chips, and MediaTek™ processors. The processor 210 may also be customized to meet specific needs of the device, such as low power consumption, high performance and / or specialized tasks like artificial intelligence processing or image processing.

[0106] The memory 220 of the handheld device can include at least one nonvolatile storage component for storing the operating system, applications, data and / or other system files. The memory can include various types of storage, including but not limited to, flash memory, NAND memory, and solid-state drives (SSDs). In addition to non-volatile memory, the device 200 may also include volatile memory, such as random access memory (RAM), to store temporary data needed by the processor during operation, including application code for execution. The RAM may be of various types, including DDR, LPDDR, and LPDDR5 RAM, with varying data speeds, power consumption, and sizes.

[0107] The handheld device 200 may include a display 230. The display 230 can be a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, a microLED display, or an e-ink display, among others. The LCD may be a thin-film transistor (TFT) LCD, in-plane switching (IPS) LCD, or twisted nematic (TN) LCD. OLED displays, such as AMOLED (Active Matrix OLED) or POLED (Plastic OLED), may be used for their ability to offer high contrast ratios and energy efficiency, with some supporting features like flexible or curved displays. The display may further support a variety of resolutions, such as HD, Full HD, Quad HD or 4k resolution. The size of the display can range for instance from a small 4-inch screen to larger 12-inch or more screens, and it may include touch capabilities, such as a capacitive touchscreen and / or a resistive touchscreen, allowing for user input via tapping, swiping and / or other gestures.

[0108] The handheld device 200 may include other or further output means, including for instance a speaker 240 capable of outputting sound to allow the user to hear notifications, calls, media and / or other audio outputs. The speaker 240 may be a dynamic speaker, piezoelectric speaker, or an electromagnetic speaker, with varying size, power consumption, and sound quality characteristics. In some embodiments, the handheld device may include multiple speakers 240 for stereo or multi-channel sound output, or may additionally or alternatively use external audio accessories such as Bluetooth speakers, wired headphones and / or a wired headset. The audio may also be processed through an integrated audio processoror sound chip that enhances sound quality, applies equalization or adjusts the audio output based on surrounding conditions.

[0109] The handheld device 200 further includes at least one position and / or movement sensor, such as an inertial measurement unit 253 (IMU), which may include at least one gyroscope 251 , at least one accelerometer 252 and / or at least one magnetometer 250. The gyroscope 251 and accelerometer 252 can be used to detect and measure the orientation, acceleration and motion of the device, and convert the same to electric signals that can be interpreted by the processor 210. The accelerometer 252 can be configured to measure linear acceleration in three dimensions, e.g., along x, y, z axes, and can be used to detect changes in movement such as tilting, shaking and / or movement in space. The gyroscope 251 can measure angular velocity, providing information about the rotational movements of the device 200 around its axes. Various types of accelerometer 252 may be employed, including capacitive accelerometers, piezoelectric accelerometers and / or microelectromechanical systems (MEMS) accelerometers. Each accelerometer or some of the accelerometers in a combination of accelerometers can have different sensitivities, ranges and / or power requirements. The gyroscope 251 can include a ME MS -based gyroscope, a fibre-optic gyroscope and / or a ring laser gyroscope.

[0110] The device 200 may also include a camera system that includes one or more image sensors, lenses and / or optical elements for capturing photographs or videos. These cameras can vary in resolution, from standard VGA to high definition or ultra-high-definition sensors, and may include features like autofocus, optical image stabilization, or depth sensing. Moreover, the handheld device 200 may also include geolocation sensors such as a Global Navigation Satellite System (GNSS) receiver, a Global Positioning System (GPS) receiver and / or other positioning sensors, configured to determine a geographic location, movement and / or orientation of the handheld device 200. Furthermore, the handheld device 200 may also include various communication modules. These include wireless communication technologies such as Wi-Fi, Bluetooth™, cellular network radios(e.g., LTE, 5G), infrared (IR) sensors, near field communication (NFC). The device may be capable of short-range communication through Bluetooth™ Low Energy (BLE) for peripherals or be equipped with Wi-Fi or future wireless standards for high-speed data transfer. The cellular module may support a variety of frequencies and protocols, allowing for global roaming and communication. Additionally, the device may include a USB interface for wired data transfer, charging, and peripheral connection, as well as an HDMI output for video streaming.

[0111] The memory 220 may include and / or the processor 210 may be configured to run a controller application 211 and / or a media application 212, the controller application 211 being configured to measure data indicative of the motion of a swing or an angular motion akin to that of a swing and to control therewith a behaviour of the media application 212. It can be appreciated that, while the controller application 211 and the media application 212 are presented as two distinct applications running in the same device 200, other arrangements are possible. In some embodiments, controller application 211 is configured or configurable to run in one device and media application 212 in a distinct device. In some embodiments, controller application 211 and media application 212 are conceptually or programmatically arranged as two modules that are part of a single application. In some embodiments, two or more applications and / or modules can be implemented to share among themselves the tasks and computations described below as being part of controller application 211 and media application 212.

[0112] The controller application 211 is configured to produce input values to control the behaviour of other applications such as media application 212. An input value can for instance correspond to a real value in range [0; 1] and can be proportional to a motion signal indicative of a relative level of kinetic energy applied to the handheld device 200, deliberately by an individual handling the handheld device 200 and / or indirectly through movements of the individual holding device 200 of having device 200 on their person. The motion signal and / or the level of kinetic energy can be computed using any suitable sensor(s) of handheld device200, for instance one or more gyroscopes 251 configured to measure angular velocity. Although a means of computing an energy level based on gyroscope readings is detailed below, it can be appreciated that other sensor(s) can additionally or alternatively be used, including for instance accelerometer(s) 252 and / or magnetometer(s) 250, or that aggregated readings from multiple sensors of the IMU 253 can be used.

[0113] The controller application 211 can be configured to listen to a broadcast stream of events corresponding to motion signals and / or readings from the gyroscope(s) 251 at a configurable suitable sampling frequency, e.g., 30 Hz or 40 Hz, and, based on a series of recent motion signals and / or readings, to infer a momentum level indicative of a magnitude of angular displacement over a sliding time window of a suitable, configurable duration, e.g., 1 second. In some embodiments, each gyroscope 251 reading in a series a)t, corresponding for instance to a three-dimensional angular velocity vector a>t= {a)x t,a)y t, )z t}, indicating a rate of rotation around each of three axes x,y,z, can be compared to the precedent reading a)^ =todetermine a level of change in momentum between times t and t - 1. The changes between readings over the sliding time window can be aggregated, for instance, by averaging the distances d, where w is the duration of the time window and f is thefrequency, for instance, respectively in seconds and in hertz, with, as examples," " " " "In some embodiments, the momentum level can alternatively or additionally be determined based on changes in rotation direction, or reversals. As examples, the momentum level can take into account the frequency of reversals and / or the angular acceleration during the reversal.

[0114] In some embodiments, at least some sensing functions and optionally some computing functions of controller application 211 can be performed by equipment installed on the physical engagement apparatus, e.g., one or more infrared sensors installed on a swing to detect passages of a user on a swingplatform or on a stationary bicycle to detect passages of a foot of the user on a pedal to infer the momentum level. In some embodiment, sensors and / or sensor modules provided with the equipment can be integrated with electronic components such as a printed circuit board, a microcontroller, a system-on-chip, a small general-purpose computer and / or other embedded computing hardware, and can be configured to perform signal acquisition, preprocessing, aggregation, local computation and / or signal transmission. In some embodiments, the equipment can comprise an Internet-of-Things (loT) device or node configured to communicate sensed data and / or derived metrics to an external device or system, such as a nearby gateway and / or a remote server, and / or a directly to a handheld device executing controller application 211 and / or media application 212, via wired and / or wireless communication means including, without limitation, Bluetooth™, Wi-Fi™, cellular communication, and / or other short-range or long-range networking protocols. In some embodiments, such communication can be unidirectional or bidirectional and can support real-time, near-real-time and / or asynchronous data exchange, configuration, calibration and / or firmware updates. In some embodiments, configurable infrared sensors can be provided with the equipment, such that, for instance, a power level and / or wattage of an infrared emitter can be configured to adjust a sensing range of the sensor and thereby enable deployment of the sensor in association with different types of physical engagement apparatus.

[0115] The controller application 211 can be configured to apply one or more transformations to the momentum level, momentum change level and / or the aggregation thereof before using the same to control the media application 212 or another application. In some embodiments, the momentum level, momentum change level and / or the aggregation thereof is clamped by suitable bounds, e.g., [0; l], In some embodiments, the momentum level, momentum change level and / or the aggregation thereof is reduced by a suitable small quantity E, e.g., E = 0.2, to avoid inferring a controller input from small momentum changes that can correspond to noise such as gyroscope drift, such that, for instance, corrected value x£= x - E or xE= max(x - E, 0). In some embodiments, a suitableinterpolation function is used to make the distribution of momentum levels, momentum change levels and / or the aggregations thereof non-linear, such as a smoothstep interpolation function, e.g., Sx(x) = 3x2- 2x3orS2(x) = 6%5- 15x4+ 10x3for % e [0; l]. In some embodiments, the controller application 211 is configured to apply a configurable difficulty factor to the momentum level, momentum change level and / or the aggregation thereof to configure the level of momentum or momentum change required to produce a noticeable effect in the controlled application. As an example, a factor of 4 can correspond to the easiest difficulty level and a factor of 1 or below 1 but above 0 can correspond to the hardest difficulty level. It can be appreciated that variations can be applied in any suitable order. As an example only, the momentum level, momentum change level and / or the aggregation thereof can be first multiplied by the difficulty factor, then reduced by E, then clamped to [0; 1], and finally smoothed by S2.

[0116] While controller application 211 is suited to translating motion created by an individual using a swing as input for applications such as media application 212, it can be appreciated that different alternative types of motions of the individual or caused by the individual can be translated as input. As examples only, the individual can control an application by vigorously shaking handheld device 200, or by performing various activities while holding or having on their person, e.g., in a pocket or in a waist or armband pouch, handheld device 200, including for instance using playground equipment other than a swing, such as a seesaw, walking, running or riding a human-powered vehicle such as a bicycle, by performing exercises, such calisthenic exercises, including for instance jumping jacks, jump squats, burpees, or on exercise and / or leisure equipment, such as a stationary bicycle as described below.

[0117] The media application 212 can be configured to render an audio arrangement such as a musical composition comprising multiple individual tracks, such as stems, layers, channels and / or tracks, for instance through the speaker 240 and / or through an external sound system installed on a site, e.g., in a public installation such as a swing apparatus 100, connected to the handheld device 200through a suitable wired or wireless communication means such as a TS, TRS or TRRS audio cable, Bluetooth™ and / or Wi-Fi™, based in input received from a controller such as controller application 211. For instance, the musical composition or a track thereof can be rendered with playback parameters such as loudness, tempo, pitch and / or tone colour configured by the input provided by the controller. As an example, if the media application 212 received an input indicative of a higher kinetic energy level by the controller application 211 , the composition or track can be played louder and / or faster. It will be appreciated that, where reference is made to musical compositions in the present disclosure, other audio arrangements including as examples other types of audio performances such as storytelling, spoken narration, sound effects and / or combinations thereof with or without music.

[0118] As an example, the audio arrangement can comprise narrated content such as fictional stories, fairy tales, myths, legends, interactive narratives, audio drama and / or role-playing scenarios, which can be rendered for individual users or groups of users and can optionally include background music, ambient soundscapes and / or synchronized sound effects. In some embodiments, the audio arrangement can comprise non-fiction content such as documentaries, historical accounts, guided tours, museum explanations, educational lessons, languagelearning exercises, scientific explanations and / or instructional material, optionally adapted in real time based on user interaction, motion, location and / or contextual parameters. In some embodiments, the audio arrangement can comprise meditative content, relaxation exercises, guided breathing sessions, mindfulness instructions and / or therapeutic audio, optionally combined with slowly evolving musical layers or environmental sounds. In some embodiments, the audio arrangement can comprise game-related audio such as dynamic soundtracks, spatialized effects, narrative prompts, challenges and / or feedback cues that evolve based on user behaviour, group dynamics and / or physical interaction with the installation.

[0119] In some embodiments, the audio arrangement can further comprise procedural or generative audio content, wherein musical elements, spokensegments and / or sound effects are selected, combined, modified and / or sequenced dynamically in response to input signals received from the controller application 211. For instance, movement patterns, acceleration, orientation, user count, proximity between users and / or temporal patterns can influence the selection of narrative branches, the intensity or density of sound effects, the pacing of spoken narration and / or the complexity of musical layers. In some embodiments, the same installation can alternate between or concurrently provide different types of audio content, such as playing music during a first interaction phase and transitioning to narrated or educational content during a second interaction phase, without limitation.

[0120] Accordingly, the media application 212 together with the controller application 211 can support a wide range of audio-based experiences for public or private interactive installations, including entertainment, education, artistic expression, wellness, accessibility and / or information delivery, whether for individual listeners, small groups or large audiences, and whether rendered locally, distributed across multiple speakers or synchronized across multiple devices.

[0121] With additional reference to figure 2C, an exemplary system for rendering an audio arrangement such as a musical composition associated, e.g., with a swing apparatus is shown. In some embodiments, the media application 212 is configured to receive inputs from a plurality of controllers, for instance operated by a plurality of players 50, 50a, 50b. Each player can hold or have on their person a distinct handheld device 200, 200a, 200b, each programmed with controller application 211, and thereby each configured to provide an input indicative of a kinetic energy level to the media application 212. In some embodiments, each controller is associated with one or more individual tracks of an audio recording such as a musical composition, with the motion of each player thereby being used to control playback parameters for one or more individual tracks. As an example, the audio track corresponding to a musical composition may include a plurality of instrument tracks, each having their playback controlled by a different player.

[0122] The media application 212 can be implemented to leverage publicly accessible infrastructure such as swing apparatus 100, which may include preexisting infrastructure. As an example, one or more musical composition can be associated with one or a number of apparatus, for instance via a machine-readable code affixed on apparatus or via a database server accessible by handheld devices 200 providing mappings of coordinates, e.g., GPS coordinates, to information related to an apparatus located at or near the coordinates. In some embodiments, the media application 212 is configured to allow the user to capture an image of an apparatus or a portion of an apparatus such as a swing apparatus through an image sensor of handheld device 200, decode a machine-readable code such as a QR code affixed to the apparatus, obtain via an internal and / or external database details of one or more audio recordings associated with the specific apparatus via the machine-readable code, and wait for input from one or more motion-based controllers such as a controller application 211 being executed on the handheld device 200 and / or on one or more nearby handheld device(s), such as devices 200a, 200b, to parametrize the playback of the audio recordings. In some embodiments, approaches such as the ones described above can be combined. As an example, the media application 212 can be configured to capture and decode an image of a machine-readable code and obtain geographic coordinates, and, through for instance an internal and / or external database, verify that the machine-readable code effectively corresponds to a defined geographic zone the coordinates are associated with, thereby implementing geofencing. Geofencing can advantageously be used, for instance, to ensure that licences on the audio arrangements are obeyed. Nearby devices 200 can discover one another and exchange data using various wireless communication technologies such as Bluetooth™, Wi-Fi™ Direct, NFC (Near Field Communication), and Ultra-Wideband (UWB). In some embodiments, one of the devices 200 is designated as the master device and is configured to receive input from the controller application 211 running on nearby devices and to render the musical composition(s).

[0123] This creates opportunities for individuals to connect with their surroundings and with others, allowing them to explore their local environment and fostering serendipitous encounters and meaningful interaction.

[0124] The instructions that enable a handheld device 200 to perform a given method or function, including the functionalities of controller application 211 and media application 212, can be stored in the form of a computer program. This computer program may be implemented in a high-level programming language, such as an imperative language, including procedural or object-oriented languages like C++, Java, Python or Dart, which are suited for a wide range of applications and can easily interface with various system components. High-level programming languages can also include declarative languages, such as functional languages like Haskell or logic languages like Prolog, which allow developers to specify what the program should accomplish rather than describing step-by-step operations. These high-level languages can improve development efficiency and code readability.

[0125] Alternatively, computer programs may be implemented in low-level languages, such as assembly or machine code, especially when direct hardware control or optimization is required. Low-level languages are closer to machine instructions and provide precise control over hardware resources, which can be advantageous in resource-constrained environments, such as embedded systems. Programs written in low-level languages can be used in applications that require high performance, small memory footprints, or real-time processing capabilities.

[0126] Each computer program may be either compiled or interpreted. Compiled languages, such as C, C++ or Dart, can be transformed into machine code optimized for a specific hardware configuration, allowing efficient execution. Compilation can result in highly optimized executables that are tailored to the underlying architecture, which is advantageous in performance-critical applications. Interpreted languages, such as Python or JavaScript, offer flexibilityby interpreting code at runtime. This allows for rapid development and platform independence, as the same code can be run on different systems with minimal modifications. Hybrid approaches, such as Java bytecode or .NET Common Intermediate Language (CIL), combine elements of both compiled and interpreted paradigms. In these cases, code is compiled to an intermediate representation that can be executed by a virtual machine on various platforms, providing crossplatform compatibility.

[0127] The computer programs can be designed to leverage various resources provided by the operating system of the handheld device 200, such as Android™ and iOS™, to enhance functionality, performance, and user experience. These resources include but are not limited to, operating system libraries, APIs, and frameworks that offer access to a wide range of hardware and software capabilities. For instance, on Android™ devices, libraries such as Android™ SDK provide essential resources like View Frameworks™, Location Services™, and Sensor Frameworks™ for interacting with devices’ GPS, accelerometers 252, gyroscopes 251, and other sensors. On iOS™, frameworks such as Core Motion™, Core Location™, and UlKit™ offer similarly capabilities for handling device sensors, user interface elements, and geolocation features. Additionally, the application may utilize Multimedia Libraries such as OpenGL™ or Metal™ for rendering graphics, MediaPlayer™ for audio and video handling, and / or TensorFlow™ for machine learning tasks. By leveraging these built-in libraries, the applications 211, 212 are able to perform tasks such as device motion detection, image and video processing, real-time data synchronization, and seamless interaction with other applications or system features. The use of these libraries can significantly reduce the need for custom development and optimize the applications’ ability to run efficiently on diverse device 200 models and operating system versions.

[0128] With reference to figure 3, an exemplary method 300 for rendering a musical composition associated with a physical engagement apparatus is provided.

[0129] In a first step 310, a first individual acquires, using the camera of a handheld device running a suitable application, hereafter called the media application, a machine-readable code such as a QR code affixed to apparatus.

[0130] In a subsequent step 320, the media application obtains from an internal and / or an external database an audio arrangement such as a musical composition associated with the swing apparatus via the machine-readable code. The musical composition can include a number of individual tracks, for instance instrument tracks.

[0131] In simultaneous or subsequent step 330, in a multi-user scenario, the media application can detect and pair with nearby handheld devices of additional individuals running a controller application configured to control the rendering of the audio arrangement by the media application. The handheld device of the first individual can also run the controller application and participate in controlling the rendering.

[0132] In subsequent step 340, the media application can associate each controller application running on a paired device or on the same device with one or more individual tracks of the audio arrangement. In some embodiments, the assignment is performed such that each track is assigned to one controller application.

[0133] In subsequent step 350, the media application acquires a motion-based input from each controller application and, in step 360, renders the musical composition by rendering each track based on the input acquired from the associated controller application. The process of steps 350 and 360 is repeated until the whole musical composition has been rendered.

[0134] With reference to figure 4, an exemplary physical engagement apparatus 400 corresponding to a generic stationary bicycle is illustrated, along with a user 50 interacting with the bicycle during operation.

[0135] The stationary bicycle 400 includes a frame 410, which provides structural support and maintains the overall integrity of the bicycle. The frame 410 can be constructed from various materials, including but not limited to metal alloys, reinforced plastics, and composite materials. In some embodiments, the frame 410 includes an adjustable seat 420, which can be secured to the frame using a seat adjustment mechanism 422, allowing the user 50 to modify the height and / or horizontal position for comfort and optimal pedalling efficiency.

[0136] The stationary bicycle 400 can include handlebars 430 positioned in front of the user 50 and can be rigidly affixed to the frame 410 or configured with an adjustment mechanism 432 to allow modifications in height, tilt, and / or reach. In some embodiments, the handlebars 430 include additional components such as integrated controls, sensors, or grip-based user input mechanisms.

[0137] The stationary bicycle 400 includes a drive mechanism 440 enabling the user 50 to engage in a pedalling motion. This drive mechanism 440 can include pedals 442, which can for instance be coupled to a crankset, e.g., including crank arms 444, transmitting rotational motion, e.g., to a flywheel 450 via a belt or chain drive 448. The flywheel 450 can be designed to provide inertia and simulate the feel of outdoor cycling. In some embodiments, the stationary bicycle 400 can be configured as a stationary “handcycle” in which the pedals 442 can be placed and configured to allow pedalling with the hands of the user 50 in addition or as an alternative to the feet. A resistance mechanism 452 can be operatively connected to the flywheel 450 to modify the difficulty of pedalling. The resistance mechanism 452 can employ various technologies, including but not limited to magnetic resistance, friction pads and / or air-based resistance.

[0138] The stationary bicycle 400 can include a sensor module 460 configured to monitor and / or analyze the pedalling activity of the user 50. The sensor module 460 can include a speed sensor to measure rotational velocity, a cadence sensor to track pedalling rhythm and / or a power output sensor to estimate energy expenditure. Additionally or alternatively, the stationary bicycle 400 can beprovided with an infrared sensor as described above configured to detect passages of at least one foot of the user 50 and thereby infer or refine the computation of a pedalling rhythm. Additionally or alternatively, the stationary bicycle 400 can be provided with an imaging sensor module configured to acquire and analyze an image of at least one foot of the user 50 to infer or refine the computation of a pedalling rhythm. The sensor module 460 can thereby function as a motion sensor, e.g., a sensor configured to generate a control signal in response to the user 50 actuating the crankset, based on a kinetic energy level associated with the actuation. As examples, the kinetic energy level can be computed based on the rotational velocity, the pedalling rhythm, the energy expenditure, the difficulty of pedalling, e.g., induced by the resistance mechanism 452, or any combination thereof. As examples, the speed and / or cadence sensor can include an optical encoder, Hall effect sensor and / or magnet-based reed switch. In some embodiments, additional sensors can be integrated, such as force sensors detecting pedal pressure, inclination sensors for user posture assessment, or biomechanical sensors capturing limb movement. These sensors can communicate data through wired or wireless connections to an external system, including but not limited to a gaming console, a computer, or a virtual reality system.

[0139] In some configurations, the stationary bicycle 400 is adapted for use as a controller in interactive applications, including video games, simulations, and fitness programs. A system and / or an apparatus to adapt the stationary bicycle 400 for use as a controller can be designed for compatibility with generic, off-the-shelf stationary bicycles by incorporating external sensor attachments, or it can be implemented in purpose-built exercise apparatus with integrated electronics. Variations in the arrangement and functionality of these components can be applied to accommodate different user needs, preferences, and technical constraints.

[0140] With reference to figures 5A to 5C, an example tilt-based controller system 500 is illustrated. Broadly described, the system includes a physical engagementapparatus 400, adapted to function as a tilt-based controller, in active use by a user 50 and an image sensor 510 connected to an electronic entertainment device 520 configured to render a virtual environment on a display 530 based at least in part on readings of the image sensor 510.

[0141] The system 500 includes an electronic entertainment device 520. The entertainment device 520 can include any suitable processing device configured to execute interactive software and render a virtual environment, i.e. , a computer-generated, interactive simulation of a bidimensional or tridimensional space, possibly including one or more avatars, which can correspond to graphical representations of users or characters within the environment, based on control signals received from one or more input devices such as controllers. In some embodiments, the entertainment device 520 can be a home video game console such as a Nintendo™ Switch™, a PlayStation™ and an Xbox™ console, which can be connected to a television, computer monitor, projector or other suitable external display 530, to which the environment can be rendered. In some embodiments, the entertainment device 520 can be a personal computer, such as a laptop or desktop computer, operatively connected to a monitor or similar display 530. In other embodiments, the entertainment device 520 can be a mobile device, such as a smartphone, tablet or handheld gaming console, having an integrated display 530. In some embodiments, the entertainment device 520 can be a smart television or a set-top box configured to run video game software locally or stream interactive content via a network connection. In some embodiments, the entertainment device 520 can correspond to or include a head-mounted display system such as a virtual reality (VR) or augmented reality (AR) headset. The VR or AR system can integrate both rendering hardware and a display 530. The entertainment device 520 can receive control signals from a controller, and can be configured to use those signals to update or modify the rendering of the virtual environment, such as adjusting the position, orientation and / or speed of an avatar or vehicle rendered within the environment. The entertainment device 520 can also include audio and / or haptic feedback systems, and in some embodiments cansupport networked gameplay or remote data synchronization with cloud-based platforms.

[0142] The system 500 includes a physical engagement apparatus 400 acting as a controller or one of the controllers of the electronic entertainment device 520, i.e., an equipment which the user 50 interacts with to generate control signals directed to the electronic entertainment device 520. In some embodiments, the physical engagement apparatus 400 is a stationary bicycle, and control signals can include or be based on parameters including, for example, cadence, resistance level, button presses and / or other user interactions detected on the bicycle. While parameters such as cadence and / or resistance level are particularly suitable, as an example, to control forward movements of an avatar or a vehicle, it can be appreciated that they do not offer an inherent way of efficiently controlling for lateral movements of the avatar or vehicle.

[0143] There is a need, therefore, for a controller capable of generating control signals suitable for lateral movement. The system 500 includes an imaging sensor 510 configured to acquire image data of the user 50 while the user interacts with the stationary bicycle or other physical engagement apparatus 400, including for instance one or more photographic image(s) of the user 50 and / or a video of the user 50. The system is configured to generate control signals based on the pose of the user 50 detected in the image data, for instance a pose corresponding to a tilting direction and / or degree of the upper body of the user 50, thereby defining a tilt-based controller. Advantageously, this makes it possible to generate control signals suitable for lateral movements with generic physical engagement apparatus 400, for instance a generic stationary bicycle, and without requiring specialized hardware other than the imaging sensor 510. Furthermore, it can be appreciated that this approach is equally compatible with other types of equipment, including as examples only a rower, a stability ball, a foam roller, a slant board and / or a hyperextension bench.

[0144] In some embodiments, the imaging sensor 510 can be a purpose-built camera module designed specifically for fitness or gaming applications. Such a purpose-built imaging sensor 510 can be mounted in a variety of locations, including on a structural portion of the stationary bicycle 400, on a nearby stand, on a wall, on the ceiling or integrated into a dedicated accessory such as a light bar or sensor bar. In other embodiments, the imaging sensor 510 can be part of a general-purpose device comprising an integrated camera, such as a smartphone, tablet, laptop or webcam, positioned in any suitable manner to capture an image of the user during operation. The imaging sensor 510 can be secured using a clamp, tripod, suction mount, magnetic mount, articulated arm or other adjustable fixture that allows optimal positioning relative to the user. In some embodiments, the imaging sensor 510 can be integrated with a smart display, such as a smart mirror or smart TV, which can also serve as the display 530 of the entertainment device. The imaging sensor 510 can include one or more optical elements, infrared sensors, depth sensors, structured light projectors and / or stereoscopic cameras to enable the capture of two-dimensional and / or three-dimensional image data.

[0145] The captured image data can be processed locally, and / or transmitted to a part of the system 500 such as the electronic entertainment device 520 and / or to a remote server for analysis. In some embodiments, a pose estimation module, i.e. , a module for estimating a human pose from image data, can process input images or video frames to detect and track key points corresponding to anatomical landmarks of a subject. The system 500 and / or the remote server can implement one or more machine learning models trained to identify anatomical landmarks corresponding to locations of body joints and / or skeletal features, such as shoulders, elbows, wrists, hips, knees and ankles, based on visual cues extracted from the image data. In some embodiments, the machine learning models can be trained to also determine additional key points, such as facial landmarks, hands, feet and / or torso features, to enhance the accuracy of the pose estimation. The detection can be performed using a deep neural network, such as a convolutional neural network (CNN), a transformer-based model, a graph-based model, and / or a hybrid approach that incorporates multiple network architectures.

[0146] In some embodiments, the tilt-based controller can correspond to a handheld device integrating both the imaging sensor 510 and the pose estimation module. In some embodiments, the tilt-based controller can include both an imaging sensor and separate computational components configured to implement the pose estimation module.

[0147] In some embodiments, the pose estimation module can employ a multistage process, where one or more first model(s) detect(s) the presence and approximate location of a person in an image, and one or more second model(s) refine(s) key point locations by analyzing spatial relationships between detected landmarks. As examples, the initial detection can be performed using an object detection network, a heatmap-based key point regression approach, a single-shot key point estimator, and / or a combination of these techniques. As examples, the refinement process can utilize probabilistic graphical models, attention mechanisms, geometric constraints, and / or temporal smoothing techniques to improve consistency and accuracy. In some embodiments, the pose estimation module can incorporate prior information regarding human body proportions, motion dynamics and / or biomechanical constraints to refine predictions and filter out implausible poses.

[0148] In some embodiments, the pose estimation module can generate a structured representation of the detected pose, such as a skeletal model, a 2D or 3D pose graph, and / or a set of numerical descriptors that encode joint positions and / or angles. The system can also perform real-time tracking by associating key points across consecutive frames, which can involve optical flow estimation, recurrent neural networks (RNNs), Kalman filtering, particle filtering, and / or other state estimation methods.

[0149] In some embodiments, the system can be optimized for different operational constraints, such as low-power processing environments, high-accuracy applications, and / or real-time responsiveness. The model can be quantized, pruned and / or optimized using hardware acceleration techniques, suchas tensor processing units (TPUs), neural processing units (NPUs), graphics processing units (GPUs) and / or field-programmable gate arrays (FPGAs). The system can also adapt dynamically to input conditions, such as varying lighting, occlusions, camera angles and / or subject movement, using domain adaptation, self-supervised learning, transfer learning, and / or active learning techniques. In some embodiments, the models can operate in conjunction with other sensing modalities, such as depth sensors, inertial measurement units (IMUs), radar and / or LiDAR, to enhance pose estimation accuracy and robustness in challenging environments.

[0150] In some embodiments, the pose estimation module can be configured to detect a suitable but limited number of anatomical landmarks in the upper body of the user 50, for instance two or four landmarks, and output, or generate, coordinates, for instance bidimensional coordinates of the landmarks in the reference frame of the image. The number of detected landmarks can be based on a tradeoff between a required complexity of the pose required and the availability of computational resources. For instance, detecting whether a user 50 is leaning or tilting their upper body, the direction and the degree of such a tilt can be performed with two medial landmarks, i.e. , landmarks located on or near the median plane of the body such vertebrae, at different heights. Estimating the coordinates of medial landmarks in the image frame can be done directly and / or with reference to two or more peripheral landmarks, i.e., landmarks located away from the body’s centre. This can make it possible to generate the required control signal by detecting between two and four landmarks, thereby requiring less computational resources such as memory and / or processing time and making the process manageable, for instance, by a smartphone.

[0151] In some embodiments, the pose estimation module can be configured to detect a number of primary anatomical landmarks from the image, and to use the primary anatomical landmarks to compute, or generate, the locations, e.g., coordinates, of secondary anatomical landmarks based on the locations of the primary anatomical landmarks.

[0152] As an example, four detected, or primary, anatomical landmarks can correspond to the two acromioclavicular joints 51, 52 and the two femoroacetabular joints 54, 55. A line segment between the estimated coordinates of the left acromioclavicular 51 joint and the right acromioclavicular joint 52 can define a centre point 53, which can be used as a secondary anatomical landmark, with coordinates approximately corresponding to coordinates of an upper thoracic vertebra, e.g., T2 of the user 50. A line segment between the estimated coordinates of the left femoroacetabular 54 joint and the right femoroacetabular joint 55 can define a centre point 56, which can be used as a secondary anatomical landmark, with coordinates approximately corresponding to the coordinates of the lumbosacral joint of the user 50. It can be appreciated that the line segment 57 between the centre points 53 and 56, therefore, extends along the general orientation of the vertebral column of the user 50. An angle of line segment 57, for instance the oriented angle between segment 57 and a reference axis such as a reference straight line extending vertically from the centre point 56 along the y-axis of the reference frame, can be used to quantify the tilting direction and degree of the upper body of the user 50. In some embodiments, the angle corresponds toare the coordinates of point i. In someembodiments, the angle serves as one of the control signals input by entertainment device 520. In some embodiments, detected landmarks, i.e. , primary anatomical landmarks, can be used directly to compute an angle, such that the primary anatomical landmarks are also considered to be secondary anatomical landmarks. In some embodiments, one of the control signals input by entertainment device 520 is based on the angle.

[0153] Although the illustrated embodiment uses at least in part upper body tilt of the user 50 to generate control signals, it can be appreciated that, in some embodiments, other poses and / or visual cues detectable in the image data can alternatively or additionally be used to generate such control signals. For example, the pose estimation module can detect and respond to gestures such as raising one arm, extending both arms laterally, rotating the torso, bending the knees,shifting body weight from one side to another, or performing a specific sequence of limb movements. In some embodiments, the detection of a hand wave, a clenched fist, a pointing gesture, or a change in head orientation such as nodding or turning the head can be used to generate or modify control signals. The system can also respond to changes in spatial relationships between key points, such as the distance between the hands, the angle formed by an elbow joint, or the symmetry of limb positions. Additionally or alternatively, control signals can be generated based on temporal patterns in pose changes, such as rhythmic motion, sudden transitions, or sustained postures maintained over a threshold duration. In some implementations, contextual factors such as the detected orientation of the user’s body with respect to the camera, the presence of occlusions, or the position of the user 50 within the frame can also influence or modulate control signal generation. Accordingly, a wide variety of pose-based features, individually or in combination, can be leveraged.

[0154] In some embodiments, pose-based control signals as described above can be combined with additional input from other sensors or controllers, including for instance one or more generic controllers such as a gamepad, a joystick and / or a racing simulation controller, and / or one or more sensors used to implement an exercise apparatus-based controller. As an example, in some embodiments, the sensor module 460 of the stationary bicycle 400, can be used as a controller for the entertainment device 520, by being configured to translate sensor readings into control signals and / or by being configured to transmit sensor readings to an external device such as a mobile device, for instance imaging sensor 510, entertainment device 520 and / or a different device, which is configured to translate the sensor readings into control signals. As explained above, the sensor module 460 can include one or more sensors configured to measure parameters such as pedal cadence, crank angle, pedal force, torque, acceleration, direction of rotation, and / or resistance level. The measured values can be sampled continuously or periodically and processed to generate control signals indicative of the user’s cycling behaviour. These control signals can be interpreted as input for controlling aspects of interactive content, such as the speed of an avatar or of a virtual vehicle.

[0155] In some embodiments, imaging sensor 510 or a device including it and / or any other device acting as a controller such as, in some embodiments, sensor module 460, is operatively connected to the entertainment device 520 using any suitable communication interface. The connection can be established through a wired interface, such as USB™, HDMI™, Ethernet or a proprietary data cable, or through a wireless interface, such as Bluetooth™, Wi-Fi™, Zigbee™, cellular, infrared or other short- or long-range wireless protocols. In some embodiments, the controller device can communicate directly with the entertainment device 520, while in other embodiments the communication can occur indirectly via one or more intermediary devices, such as a smartphone, a dedicated hub, a smart home assistant or a local area network router. In some implementations, the controller device can pair with the entertainment device 520 through a standard device discovery process or by scanning a QR code, tapping via Near Field Communication, entering a pairing code and / or using other ad hoc pairing techniques. In some embodiments, the connection can be unidirectional or bidirectional. In some embodiments, the connection can support streaming of control signals, sensor data, video, audio, firmware updates and / or synchronization metadata. In some embodiments, a controller can be integrated into the same housing as a device that is also running part or all of the entertainment software, e.g., a smartphone or a handheld gaming device, while in other cases the controller can be a separate peripheral that interfaces with the entertainment device 520 over a network. The communication protocol used can be standardized or proprietary, and in some embodiments can be optimized for low latency, high bandwidth, low power consumption and / or resilience to interference.

[0156] In some embodiment, the imaging sensor 510 or a different and / or additional imaging sensor can additionally or alternatively be located at a position adapted to capture one or more images of at least one of the pedal 442, crank arm 444 and / or foot and / or leg of the user 50. The camera can be coupled to a hardware and / or software module configured to measure parameters such as a speed and / or frequency of rotation, e.g., in rotations per minute, based on theimages. In some embodiments, a computer-readable mark such as a sticker of a specific or distinctive colour can be affixed, e.g., to at least one of the pedal 442 and / or crank arm 444 to facilitate the measuring of the speed and / or frequency of rotation. In some embodiments, the pose estimation module can be adapted to locate and / or track anatomical landmarks, e.g., of the foot and / or leg of the user 50 to facilitate the measuring of the speed and / or frequency of rotation. In some embodiments, the additional or alternative imaging sensor and / or the hardware and / or software module can be coupled to a computing device such as a handheld device like a tablet or a smartphone and send the measured parameters to the computing device, for any suitable purpose such as storage, analysis, display and / or sharing with other individuals and / or with relevant applications such as fitness trackers. In some embodiments, the additional or alternative imaging sensor and / or the hardware and / or software module can additionally or alternatively be coupled, directly or indirectly, to an electronic entertainment device 520 and provide for instance additional or alternative control signals. In some embodiments, the additional or alternative imaging sensor and / or the hardware and / or software module can additionally or alternatively be coupled, directly or indirectly, to a metacontroller as described below and provide for instance additional or alternative signals used by the metacontroller to apply forwarding logic.

[0157] With reference to figure 6, a method 600 for operating a tilt-based controller is shown. Broadly described, the method includes capturing an image of a user 610, detecting their articulations 620, computing their pose 630, and actuating the render of an avatar in a virtual environment 640. Method 600 can for instance be implemented using the tilt-based controller system described above and illustrated in figure 5A, although it can be appreciated that other systems capable of performing the method steps can equally be used.

[0158] Method 600 includes a first step 610 of acquiring at least one image of a user while they are using an exercising equipment in the context of controlling an electronic entertainment device such as a video game or a VR system. In someembodiments, acquiring the image can include capturing one or more frames of image data using an imaging sensor, for instance positioned to have an unobstructed view of the user during exercise. The imaging sensor can be a standalone purpose-built camera or an integrated component of a general-purpose device, such as a smartphone, tablet, laptop or smart display. The image can include visible light data, infrared data, depth data or any combination thereof. In some embodiments, the imaging sensor can operate in real time to continuously or periodically capture image data while the user is interacting with the physical engagement apparatus 400. The acquired image or images can include a frontal, side, overhead and / or rear view of the user, depending on the placement of the imaging sensor. In some embodiments, multiple imaging sensors can be used simultaneously to capture different angles or to enable stereoscopic or volumetric reconstruction. Step 610 or a subsequent step can include preprocessing operations such as lens correction, noise filtering, contrast enhancement and / or resolution adjustment.

[0159] Method 600 includes a subsequent or concomitant step 620 of detecting articulations of the user in the image(s). This step can involve applying one or more computer vision techniques to the acquired image data to identify positions of anatomical landmarks corresponding to the user’s joints, such as the knees, hips, ankles, shoulders, elbows, wrists and / or neck. In some embodiments, step 320 can include using a pose estimation algorithm, for instance, including a bidimensional or tridimensional skeletal tracking model, to infer the spatial configuration of the user’s body from the image(s). The detection and / or tracking can be performed locally in the imaging sensor, in a mobile device including the imaging sensor and / or in the entertainment device, and / or remotely, for instance, via a cloud-based service. In some implementations, machine learning models trained on large datasets of human poses can be used, which can allow an increase in robustness, e.g., across different lighting conditions, user body types, camera angles and clothing styles. In some embodiments, step 320 can involve identifying multiple articulations over a sequence of images to track the user’smovements over time. The detected articulations can be represented as joint coordinates, vectors or other geometric descriptors.

[0160] In some embodiments, a limited number of articulations, joints and / or landmarks can be detected, advantageously limiting the use of computational resources, which can be particularly beneficial if the detection is performed in a device with constrained processing and memory resources such as a smartphone. In some embodiments, only two landmarks can be detected. As an example, the two landmarks can include a landmark that approximates the location of a vertebra such as an upper thoracic vertebra, e.g., T2, for instance the sternal notch, and a landmark that approximates the location of a vertebra at a different height such as a lower lumbar or a sacral vertebra, for instance the lumbosacral joint. In some embodiments, two or more landmarks can be computed based on detecting a higher number of landmarks in the image(s), e.g., three or four. As an example, an upper landmark can be computed based on detecting the left and right acromioclavicular joints and / or the left and right shoulders, and / or a lower landmark can be computed based on detecting the left and right femoroacetabular joints and / or the outer edges of the hips. In some embodiments, a single landmark can be detected, for instance, the head.

[0161] Method 600 includes a subsequent or concomitant step 630 of computing or estimating a pose of the user based on the detected landmarks. The pose computation or estimation can be performed locally in the imaging sensor, in a mobile device including the imaging sensor and / or in the entertainment device, and / or remotely, for instance via a cloud-based service. In some embodiments, step 630 includes computing a pose of the user’s vertebral column. This can for instance include relying on coordinates of an upper and a lower medial landmarks, e.g., landmarks indicating an approximate location of T2 and of the lumbosacral joint, to obtain a line segment that extends along the general orientation of the user’s vertebral column. This makes it possible to estimate an angle of the vertebral column with a reference line, for instance, a vertical line extending across the image frame. In some embodiments, step 630 can further include estimatingthe coordinates of a medial landmark based on at least two peripheral landmarks detected in the image(s). As examples, an upper medial landmark can be estimated as being the centre point of a line segment connecting the two acromioclavicular joints, and / or a lower medial landmark can be estimated as being the centre point of a line segment connecting the two femoroacetabular joints. In some embodiments, an angle of the upper body of the user can be estimated based on the position of a single landmark, such as the head, in the reference frame of the image.

[0162] Method 600 includes a final step 640 of actuating an avatar and / or any other object rendered in a virtual environment by an entertainment device based on the user’s pose. In some embodiments, the pose is used to actuate the avatar along a predefined axis such as a horizontal axis, while another means of control can be used to actuate the avatar along the other axis if the virtual environment is bidimensional, or one or both of the other axes if the virtual environment is tridimensional, e.g., vertical and / or depth axes. As an example, whether the pose is indicative of the user leaning or tilting their upper body and in which direction and / or degree can be used to determine a horizontal movement of the avatar. As another example, whether a sensor of the exercise apparatus 100 is indicative of the user exercising, e.g., whether the user is pedalling on a stationary bicycle, or rowing on a rowing machine, and the energy being expended by the user can be used to determine a forward and / or backward movement of the avatar along a depth axis. In some embodiments, the mapping of exercise apparatus 100 control to virtual actions can be configurable, and / or can adapt dynamically based on user preferences, fitness level, game context and / or training goals.

[0163] With reference to figure 7, an exemplary metacontroller system 700 is shown. Broadly described, system 700 includes an electronic entertainment device 520 controlled in part by control signals that are generated by a generic controller 710 and potentially modified by a metacontroller 720 based on input received by a physical engagement apparatus 400.

[0164] The system 700 includes a physical engagement apparatus 400 and an entertainment system 520, as described above. The physical engagement apparatus 400 can include any suitable form of equipment configured to support physical exertion by a user, such as the stationary bicycle described above, or alternatively other exercise machines including but not limited to recumbent bicycles, upright bicycles, spin bikes, handcycles, balance boards, ellipticals, treadmills, rowing machines, stair climbers, steppers, strength training stations and / or hybrid machines combining multiple movement modalities. The entertainment system 520 can be as described above and can include, for example, a video game console, a personal computer, a smart TV, a set-top box, a tablet, a smartphone, a VR headset or an AR system, configured to execute interactive software and render a virtual environment. The system 700 can further include a display device 530 operatively coupled to the entertainment system 520. As noted above, the display device 530 can be an external screen such as a television or monitor, or can be integrated into the entertainment system 520, as in the case of a smartphone or head-mounted display. The display device 530 can be used to present visual feedback to the user, including graphical elements representing a virtual environment, user interface elements, performance data and / or an avatar responsive to the user’s input.

[0165] The system 700 includes at least one generic controller 710 configured to generate control signals based at least in part on user 50 interactions with one or more controls. The controller 710 can include any suitable combination of physical, touch-sensitive, motion-based, voice-activated and / or biometric controls. In some embodiments, the controls include at least a directional control and an action control. The directional control can include, for example, an analog stick, a directional pad, a thumbstick, a joystick, a tilt sensor, a gyroscope, an accelerometer, a trackball and / or a touch-sensitive surface configured to detect directional gestures such as swipes or drags. The action control can include, without limitation, one or more buttons, triggers, keys, switches, pressure-sensitive surfaces, capacitive zones, gesture-recognition inputs, and / or voice-command interfaces. In some embodiments, the controller 710 corresponds to or includes akeyboard, which can be a physical keyboard and / or a virtual keyboard rendered on a touch-sensitive display. In some embodiments, the controller 710 corresponds to or includes a touchscreen configured to receive input via direct contact, stylus interaction and / or multi-touch gestures such as tapping, swiping, pinching or dragging. The touchscreen can also serve as a dynamic interface for displaying context-sensitive controls, menus, indicators and / or virtual joysticks or buttons, and / or serve as the display 530 onto which the entertainment device 520 is configured to render a virtual environment. In some embodiments, the controller 710 can further include additional controls such as shoulder buttons, bumpers, scroll wheels, capacitive sliders, rotary encoders, biometric readers, microphone arrays, and / or haptic feedback elements. The controller 710 can take a variety of physical forms, including but not limited to a handheld game pad, a tablet, a smartphone, a wearable device such as a glove, a wristband, an armband or a ring, a floor mat, a keyboard, a keyboard-and-touchpad combination, a driving simulator setup including, e.g., a steering wheel, one or more pedals and / or a shifter, and / or a hybrid device with reconfigurable input zones.

[0166] The system includes a metacontroller 720 configured to accept as input any data output by the exercise apparatus 400 and / or sensors of or installed on the exercise apparatus 400, for instance including sensor readings, motion signals, kinetic energy metrics such as a kinetic energy level and / or a momentum level computed over a sliding time window of a suitable, configurable duration, e.g., 1 second, and the control signals by the generic controller 710 and to output control signals for the entertainment device 520. In some embodiments, the metacontroller 720 corresponds to or includes a hardware module and can include a microcontroller, a printed circuit board assembly, and / or a general-purpose computing platform such as a Raspberry™ Pi™, an Arduino™ board, a BeagleBone™, and / or a similar embedded system. The metacontroller 720 can include one or more processors, volatile and / or non-volatile memory components, input / output interfaces, communication modules, and supporting circuitry for power management and signal conditioning. In some embodiments, the metacontroller 720 can additionally or alternatively correspond to or include a software module.In some embodiment, a single electronic device can include both sensors configured to generate a motion signal and / or an input value and the metacontroller. As an example, a single electronic device can include an infrared sensor configured to measure the engagement of the user with the physical engagement apparatus and to interface with the generic controller and / or the entertainment device 520. As an example only, in embodiments in which the entertainment device 520 is a handheld device such as a smartphone, the metacontroller 720 can correspond to or include an application configured to intercept, modify and / or inject input events. In some embodiments, the entertainment device 520 can be configured to run a metacontroller 720 application that provides a virtualized runtime environment for an entertainment application. The virtualized environment can be configured to intercept user input events, such as touch events, gestures, or keyboard inputs, prior to delivery to the entertainment application. The first application can modify, filter, or augment the input events based on the forwarding logic described below, and then transmit the resulting inputs to the entertainment application as if they were received directly from the user. The virtualized environment can be implemented using platform-specific capabilities, such as accessibility services, input method frameworks, or system-provided APIs for application sandboxing, and can be compatible with a range of mobile operating systems, including Android™ and iOS™. In some embodiments, the interception and control of inputs can be performed without requiring elevated system privileges.

[0167] Data output by the exercise apparatus 400 and / or sensors can include sensor readings indicative of whether the user 50 is actively using the exercise apparatus 400 to exert themselves, and can therefore be designated as movement signals. Broadly described, the metacontroller 720 is configured to apply forwarding logic to control signals received from controller 710 based on current and / or past movement signals, configurations and / or rules. The forwarding logic can cause the metacontroller 720, depending on the context, to transmit, or forward, the control signals unaltered, to modify them before transmission or to prevent their transmission to the entertainment system 520 altogether. The genericcontroller 710 and the exercise apparatus 400 are therefore operably connected to the metacontroller 720 and the metacontroller is operably connected to the entertainment device 520 through any suitable wired or wireless interface, including for instance USB™, HDMI™, Ethernet, a proprietary data cable, Bluetooth™, Wi-Fi™, Zigbee™, cellular, infrared or other short- or long-range wireless protocols, as described above for instance with respect to the connection of imaging sensor 510 with entertainment device 520 in the system 500 illustrated in figure 5A.

[0168] In some embodiments, the forwarding logic is adapted to provide a motivation for the user 50 to perform exercise by providing a perceived reward in the form of control signals from the controller 710 being forwarded to the entertainment device 520. In some embodiments, the metacontroller 720 is configured, if it receives movement signals indicative of a movement being detected, indicating that the user 50 is performing exercise, to forward the control signals received from the controller 710 to the entertainment device 520. The metacontroller 720 can be configured, if the movement signals stop or become indicative of no movement being detected, to cease forwarding the control signals. In some embodiments, the metacontroller 720 is configured to forward at least a portion of the control signals received from the controller 710 to the entertainment device 520, the portion being selected based on the movement signal. As an example, the metacontroller 720 can be configured to forward control signals corresponding to action controls at all times but to forward control signals corresponding to directional controls only when movement signals are indicative of movement being detected, allowing for instance an avatar to perform actions at all times but to move only while the user 50 is performing physical exercise. In some embodiments, an intensity associated with at least a portion of the control signals can be modified based on an intensity associated with the movement signals, for instance allowing an avatar to move comparatively faster when the user 50 is performing comparatively more intense physical exercise. In some embodiments, the user performing physical exercise and the user actuating the controller 710 need not be the same user. As an example, two users can team upsuch that one provides activation energy through the exercise apparatus 400 and the other controls the avatar.

[0169] In some embodiments, the metacontroller 720 is configured to accumulate data received from the exercise equipment over a period of time, for example by storing successive instances of received movement signals in a buffer, queue, memory, or other data structures. The accumulated data can then be processed collectively, or used to inform the forwarding logic. The activation of a mode in which control signals are all forwarded can therefore be delayed and occur in response to the aggregated state or content of the accumulated data. As examples, the metacontroller can be configured to forward the control signals received from the controller 710 to the entertainment device 520 continually for a set duration, e.g., one hour, once past and / or current movement signals are indicative of the user 50 having performed exercise for the same or a different set duration, and / or once stored data indicates that a suitable and configurable threshold of energy expenditure, e.g., 300 kcal, has been attained. In some embodiments, the metacontroller 720 is configured to compute statistics relative to physical activity of the user 50. In some embodiments, the metacontroller 720 is associated with an application, for instance a smartphone application or a web application configured to receive and display the statistics. In some embodiments, the application is additionally or alternatively configured to receive the raw data from the metacontroller 720 and compute at least some of the statistics. In some embodiments, the application can be configured to allow the user 50 to set and track objectives, to obtain rewards such as video game-related rewards, to join a social community formed by other users of the application, and / or to join competitions with other users of the application.

[0170] It can be appreciated that the software run by the entertainment device 520 needs not necessarily correspond to a game or to another application that is configured to render a virtual environment. As examples only, the entertainment device 520 may be configured to perform playback of music, video and / or multimedia content, for instance through websites or applications such asYouTube™, Netflix™ orTikTok™, and / or to allow for interactions between the user 50 and a conversational agent such and / or a generative model such as a large language model. In some embodiments, the metacontroller 720 is configured to authorize the performance of certain actions, such as starting the playback of a video, allowing a sound output from a device, allowing the use of a photo camera of a device or allowing a conversational agent to accept a prompt and / or to generate a response, e.g., to generate text, one or more images and / or multimedia content, based on current and / or past movement signals received from the exercise equipment. As an example, the metacontroller 720 can be configured to allow videoplayback only while receiving movement signals indicative of the user 50 performing physical exercise, and / or for a set duration based on the user 50 having performed physical exercise, e.g., for a corresponding duration and / or amounting to a specific energy expenditure.

[0171] In some embodiments, the metacontroller 720 does not require a controller and is instead configured to emulate a human interface device, for instance a USB human interface device, such as a keyboard, such that when connected to the entertainment device 520, the metacontroller 720 transmits input signals that are interpreted by the entertainment device 520 as keystrokes. The metacontroller 720 can be preprogrammed or dynamically configured to generate specific input sequences based on movement signals. As an example, a physical exercise apparatus 400 such as a stationary bicycle can include a sensor that generates a signal on a regular basis, for instance every time the crankset of the bicycle has completed a revolution, and the metacontroller 720 can emulate a specific keystroke every time the signal is received from the sensor. In some embodiments, more than one exercise apparatus 400 can be used, each given exercise apparatus 400 being associated with one given key. In these embodiments, the entertainment system 520 can run software that is configured to interpret the keystrokes in a specific way, for instance, showing more of a virtual environment or more information of display 530, for instance implementing a game that becomes more playable as the player performs more physical exercise.

[0172] With reference to figure 8, an exemplary method 800 for operating a metacontroller in the context of a video game is shown. Broadly described, when a generic game controller is actuated 810 and motion sensing on an exercise apparatus is performed 820, metacontrol logic is applied 830 to actuate the avatar 840.

[0173] The method 800 includes an initial step 810 of detecting that a user has actuated a game controller. In some embodiments, the game controller can include or correspond to any of the controller devices described herein, such as a game pad, a joystick or a steering wheel system. The actuation can include any user interaction capable of generating control signals, such as pressing a button. Detection can be performed by monitoring input signals received from the controller over a wired or wireless interface, and can include polling, interruptbased detection, signal thresholding and / or event-based triggering. The control signals and / or detected actuations are transmitted to a metacontroller.

[0174] The method 800 includes another step 820, which can precede, be subsequent to or concomitant with step 810, of sensing motion at an exercise apparatus. The exercise apparatus can include any suitable device described herein, such as a stationary bicycle or other fitness equipment configured to generate user-driven motion. In some embodiments, sensing motion can include detecting pedalling activity, linear or rotational displacement, cadence, speed, acceleration, applied force, resistance level and / or direction of movement using one or more sensors integrated into the apparatus, such as inertial measurement units, rotary encoders, strain gauges, magnetic sensors, Hall effect sensors and / or optical encoders. The sensed motion data can be captured continuously or at discrete time intervals, and can be transmitted to the metacontroller. In some implementations, motion sensing can include filtering, normalization or sensor fusion techniques to improve accuracy or robustness.

[0175] The method 800 includes a step 830, subsequent or concomitant to steps 810 and 820, of generating or forwarding control signals to an electronicentertainment device based on the motion sensed in step 820 in combination with the controller actuation detected in step 810. For instance, the control signals generated by the game controller in step 810 can be forwarded to the entertainment device if motion was sensed in step 820. If motion was not sensed, then the control signals can not be forwarded, or only a portion of the control signals can be forwarded. As an example, action controls can be forwarded, but not direction controls.

[0176] The method 800 includes a final step 840 of actuating an avatar in the virtual environment based on the control signals that are received at the entertainment device from the metacontroller. Therefore, if the metacontroller forwards direction controls, e.g., because motion was sensed, then step 840 can include moving the avatar in response to received direction controls. If, on the other hand, the metacontroller does not forward direction controls, e.g., because motion was sensed, then step 840 may not include moving the avatar.

[0177] It can be appreciated that individual features, modules, components, steps and substeps described in connection with any given system, controller, metacontroller, handheld device or method may be combined, rearranged, interchanged or otherwise integrated with features described in connection with other systems or methods to produce additional embodiments. The systems and methods described herein are designed to be modular in nature, and the functionalities of a motion-based controller, a pose-based controller, a swing-based audio system, a stationary-bicycle-based controller, and a metacontroller can be selectively merged or adapted depending on the implementation context. As an example only, certain embodiments described below may combine aspects of method 300, illustrated in figure 3, with aspects of method 800, illustrated in figure 8. For instance, a system may incorporate an audio content rendering workflow as in method 300, while simultaneously applying metacontroller-style forwarding logic as in method 800 to regulate the progression of audio, visual or interactive content based on motion detected at a physical engagement apparatus. Such hybrid embodiments may be applied to interactiveinstallations, entertainment environments, educational contexts or public-use systems, and are considered to fall within the scope of the present disclosure.

[0178] In some embodiments, a physical engagement apparatus such as a stationary bicycle can be used to control the progression of an application rendered on a suitable device including input and output modalities, such as a tablet computer affixed to, or otherwise positioned in front of, the bicycle. The tablet computer can be mounted directly onto the handlebars of the stationary bicycle, onto a structural portion of the bicycle frame, or onto a stand located in proximity to the bicycle. In such embodiments, the tablet serves as both an input and output device for an application configured to respond to the user’s pedalling activity. The user can thereby sit on the stationary bicycle, select content presented on the tablet’s display, and interact with the application through natural exercise activity without requiring additional peripherals or specialized hardware beyond the tablet and the bicycle.

[0179] In some embodiments, the application rendered on the tablet can provide access to a variety of content types. As examples only, the application can allow the user to browse or select fictional books, non-fictional books, educational material, encyclopedic entries, documentaries, interactive learning content, or audio-based entertainment such as short stories, podcasts, narrated tours, guided meditations or dramatized performances. In some embodiments, the content can additionally or alternatively include interactive or semi-interactive experiences such as simple games, quizzes, puzzle sequences, motion-gated narratives, or other digital experiences whose progression depends on continuous or intermittent user engagement through pedalling. The content can be stored locally on the tablet, streamed from a remote server, accessed from a library-managed database in the case of public installations, or retrieved through a hybrid approach combining local caching and remote lookup.

[0180] In some embodiments, prior to initiating playback of the selected content, the user can be prompted to choose a difficulty level corresponding to a pedallingintensity or speed required to sustain content progression. The difficulty level can be defined through discrete categories such as “easy,” “medium,” or “hard,” or through a more granular scale adapted to the user’s physical comfort or engagement goals. In some embodiments, the difficulty level acts as a threshold indicator such that, when the pedalling cadence remains at or above the threshold, the application continues reading aloud the selected content or continues progressing through the selected game. If, however, the user slows below the threshold or ceases pedalling entirely, the application can automatically pause playback, freeze game state progression, darken or dim the display, present a notification or encouragement message, and / or provide other appropriate feedback to prompt the user to resume pedalling.

[0181] In some embodiments, the application is configured to resume playback automatically once motion indicative of renewed pedalling is detected. In some embodiments, the application can implement more sophisticated control logic, such as gradually resuming playback speed, fading audio in or out, adjusting narration tempo, and / or resuming only after a minimum cadence has been sustained for a predetermined duration. In some embodiments, the system may provide adaptive pacing, wherein the pedalling cadence dynamically modifies parameters of the content itself, such as adjusting reading tempo, pacing animations, controlling the rate at which puzzle elements appear, or influencing the behaviour of graphical characters in a game scenario.

[0182] In some embodiments, upon completion of the selected content, the application can present the user with a summary of their session. The summary can include, as examples only, an estimate of the total distance virtually “travelled,” expressed for instance in metres or kilometres, an indication of the average pedalling intensity, a record of the time spent actively pedalling and / or metrics associated with the content itself such as reading progress or game achievements. In some embodiments, the application can store session summaries locally or remotely, allowing users to track their progress over time, compare sessions, participate in library-hosted challenges and / or share anonymized statistics withcommunity programs intended to promote physical activity or literacy. In public installations such as libraries, community centres and / or educational facilities, the system can thus provide an engaging, low-barrier-to-entry experience that seamlessly links physical activity with access to digital content.

[0183] One or more systems, methods, modules, steps or functionalities described herein may be implemented in computer programs executed on one or more processing devices, each comprising at least one processor, a data storage system (including both volatile and / or non-volatile memory and / or storage elements), and optionally at least one input and / or output device. These processing devices encompass a broad range of electronic systems capable of receiving, processing, and / or transmitting data. Examples of processing devices include, without limitation, general-purpose computers, specialized computing devices, and embedded systems. Processing devices may be implemented on dedicated hardware, including programmable hardware such as field-programmable gate arrays (FPGAs), or as software-based solutions on cloud computing platforms or serverless architectures.

[0184] Processing devices suitable for implementing the present invention may include programmable logic units, mainframe computers, servers, personal computers, laptops, cloud-based systems, personal digital assistants (PDAs), cellular telephones, smartphones, wearable devices, tablets, video game consoles, and portable video game devices. Each of these devices has the ability to execute instructions and can operate individually or in combination to perform the functionality described. The processing devices may be deployed in a variety of configurations, from single-device implementations to distributed systems that involve multiple devices collaborating to achieve a common purpose. For example, a method could be implemented on a single microcontroller in an embedded system, or distributed across a network of servers that share computational tasks.

[0185] The instructions that enable a processing device to perform a given method or function can be stored in the form of a computer program. This computerprogram may be implemented in a high-level programming language, such as an imperative language, including procedural or object-oriented languages like C++, Java, or Python, which are suited for a wide range of applications and can easily interface with various system components. High-level programming languages can also include declarative languages, such as functional languages like Haskell or logic languages like Prolog, which allow developers to specify what the program should accomplish rather than describing step-by-step operations. These high-level languages can improve development efficiency and code readability.

[0186] Alternatively, computer programs may be implemented in low-level languages, such as assembly or machine code, especially when direct hardware control or optimization is required. Low-level languages are closer to machine instructions and provide precise control over hardware resources, which can be advantageous in resource-constrained environments, such as embedded systems. Programs written in low-level languages can be used in applications that require high performance, small memory footprints, or real-time processing capabilities.

[0187] Each computer program may be either compiled or interpreted. Compiled languages, such as C or C++, can be transformed into machine code optimized for a specific hardware configuration, allowing efficient execution. Compilation can result in highly optimized executables that are tailored to the underlying architecture, which is advantageous in performance-critical applications. Interpreted languages, such as Python or JavaScript, offer flexibility by interpreting code at runtime. This allows for rapid development and platform independence, as the same code can be run on different systems with minimal modifications. Hybrid approaches, such as Java bytecode or .NET Common Intermediate Language (CIL), combine elements of both compiled and interpreted paradigms. In these cases, code is compiled to an intermediate representation that can be executed by a virtual machine on various platforms, providing cross-platform compatibility.

[0188] Each computer program implementing the methods or systems described herein is preferably stored on a computer-readable storage medium or device. Examples of such storage media include hard drives, solid-state drives, optical disks, flash memory, and magnetic tape. The computer-readable storage medium is readable by a general or special-purpose programmable computer, which, upon reading the instructions, can configure itself to perform the steps described herein. These instructions may include executable code, scripts, or markup that instructs the computer on how to operate and handle data, making the system or method functional. In some embodiments, the system or method may be embedded within an operating system running on a programmable computer, allowing for deeper integration with the hardware and enabling enhanced performance, security, or user interface features.

[0189] Processing devices implementing the present invention may contain a variety of hardware components that support program execution. Processors used within these devices include general-purpose central processing units (CPUs), which are capable of executing a wide variety of instructions, as well as specialized processors. Examples of specialized processors include graphics processing units (GPUs), which can be optimized for parallel processing and / or used in data-intensive applications like machine learning, digital signal processors (DSPs), which are designed for handling real-time audio, video, and other signal processing tasks, and application-specific integrated circuits (ASICs), which are tailored to specific functions and are often used in applications requiring high efficiency. Multicore and / or multithreaded processors can allow for concurrent execution of multiple tasks, improving overall performance, for instance in multi-user or realtime environments.

[0190] The processing device may further include various types of memory. Volatile memory, such as registers, cache, and random-access memory (RAM), can be used for temporary data storage during active program execution, providing fast access to data that the processor frequently uses. Non-volatile memory, such as read-only memory (ROM), flash memory, solid-state drives, hard disks, andoptical disks, can be used to retain data even when the processing device is powered off, making it suitable for long-term data storage. Other examples of nonvolatile storage media include diskettes, magnetic tapes, chips, and compact disks, among others. The type of memory selected can depend on specific requirements, such as the need for rapid access, data retention, or data durability under power cycling. The memory configuration of a processing device can be adjusted to support varying levels of computational demand, from lightweight applications with minimal memory requirements to complex systems requiring large data caches.

[0191] Networking solutions within a processing device enable inter-process communication and network communication over wired or wireless connections. Examples of networking technologies include Ethernet for high-speed wired connections, Wi-Fi for wireless data transmission, Bluetooth for short-range device communication, and cellular networks for broader geographic coverage. These networking solutions support various network topologies, including local area networks (LAN), wide area networks (WAN), and other network types such as personal area networks (PAN) and metropolitan area networks (MAN), as well as the Internet. Through these networks, processing devices can communicate with one another to distribute tasks, share data, and collaborate on complex computations. This communication can occur within a single building or across geographically dispersed locations, depending on the application requirements.

[0192] Implementing networking security measures can be advantageous to protect data as it travels across potentially vulnerable channels. Key security principles can include confidentiality, integrity, and availability. Confidentiality can be achieved for instance through encryption protocols like Secure Sockets Layer (SSL) and Transport Layer Security (TLS), ensuring that data remains private. Integrity can be maintained for instance with cryptographic hashing and / or digital signatures, which can detect tampering, while availability can be protected for instance by redundancy, load balancing, and defences against denial-of-service (DoS) attacks. Access control mechanisms, including multifactor authenticationand role-based access control, can be used to regulate network access. Network segmentation, such as virtual LANs (VLANs) and demilitarized zones (DMZs), can be implemented to limit access to sensitive areas and reduces the impact of breaches, while firewalls filter traffic based on predefined rules, providing an essential barrier between internal and external networks.

[0193] Advanced security measures for networking can be implemented, for instance, including encryption for wireless networks through protocols like Wi-Fi Protected Access 3 (WPA3), which can prevent unauthorized access to Wi-Fi. Intrusion detection and prevention systems (IDS / IPS) can be used to monitor network traffic for malicious activity, while virtual private networks (VPNs) can be used to establish secure connections for remote access over public networks. Regular security assessments, such as penetration testing and vulnerability scanning, identify weaknesses, and security information and event management (SIEM) systems may be leveraged to provide real-time insights into potential threats. A layered security approach, or defence in depth, can combine multiple controls across different levels of the network, enhancing resilience against both internal and external attacks by creating multiple barriers that attackers must overcome.

[0194] Distributed computing is a possible implementation in which multiple processing devices work together to perform tasks described herein. For example, a method or a method step may execute within a single thread on one processing device or be distributed across multiple threads, cores, or processors on a single device or across multiple devices. Distributed computing can help implement parallelization, where tasks are split into smaller subtasks that are processed concurrently, significantly improving processing speed and efficiency. This approach is well suited to applications with high computational demands, such as data analysis, machine learning, and large-scale simulations. In some implementations, processors are located within a single physical location, while in others, they may be spread across multiple sites, allowing for redundant and resilient computing infrastructures.

[0195] Distributed computing can also be implemented within a cloud computing environment, offering flexibility and scalability. Cloud computing architectures enable the allocation of computational resources on demand, allowing tasks to utilize as many or as few resources as needed for efficient execution. For instance, a single computational process may span multiple virtual machines, distributed across data centres in different geographical locations, to achieve optimal performance and fault tolerance. By leveraging multi-tenant architectures and dynamic scaling, cloud platforms allocate resources only as needed, reducing idle computational power. Furthermore, this approach facilitates cost efficiency, as users pay only for the resources they consume. Additionally, cloud computing’s ability to pool resources across large-scale infrastructure provides inherent redundancy and resilience, ensuring high availability for critical applications. Cloud computing can include employing containers and microservices to enhance resource efficiency and streamline deployment. Containers encapsulate applications and their dependencies in lightweight, portable units that can run consistently across different environments. This allows distributed computing tasks to be executed reliably across heterogeneous systems, reducing compatibility issues. Microservices architectures further divide applications into smaller, independently deployable services, each responsible for a specific functionality. These services can scale independently, ensuring that resources are allocated precisely where needed and minimizing waste. Together, containers and microservices enable more efficient use of computational resources, shorter deployment cycles, and improved fault isolation.

[0196] The systems and methods described herein can be distributed using edge computing architectures. Edge computing can introduce additional layers to distributed and cloud computing by bringing certain processing capabilities closer to data sources, such as sensors or devices in the industrial Internet of Things (HoT). In this architecture, certain computational tasks can be offloaded to edge devices, such as gateways or local servers, reducing latency and minimizing the volume of data transmitted to centralized data centres. This approach can be particularly advantageous in HoT applications where real-time decision-making canbe critical, such as predictive maintenance, autonomous control systems, or industrial automation. By processing data locally, edge computing can reduce bandwidth requirements, enhance data privacy, and ensure continuity of operations even when connectivity to the cloud is intermittent. This integration of edge and cloud computing can allow organizations to benefit from both localized processing and the scalability of centralized resources.

[0197] The systems and methods described herein may also be distributed in one or more computer program products, each including a computer-readable medium that bears computer-usable instructions for one or more processors. These instructions can exist in various forms, including compiled and non-compiled code, providing the flexibility needed for deployment in diverse computing environments. For example, compiled binaries may be optimized for specific hardware, while interpreted scripts or markup files can be deployed in environments where crossplatform compatibility or rapid updates are needed.

[0198] The storage and retrieval of data in a computer system may involve various data storage solutions, including relational databases, which store data in structured tables with defined relationships, and NoSQL databases, which offer more flexible storage schemas suited to unstructured or semi-structured data. Inmemory databases, which store data entirely in RAM for rapid access, can also be used in applications where low latency is desirable. These data storage solutions may be implemented on local servers, within distributed storage systems, or as part of cloud-based infrastructures, offering scalability and accessibility as required by the application. Distributed storage solutions can enable high availability and fault tolerance, ensuring that data remains accessible even if one part of the storage infrastructure fails.

[0199] Input and output devices connected to the processing device can facilitate interaction with users and other systems. Input devices can include standard peripherals, such as keyboards, mice, touchscreens, and microphones, as well as specialized input devices, such as biometric scanners, cameras, and sensors forcapturing environmental data. Output devices may encompass monitors, printers, speakers, projectors, and other display systems that present information to users in various formats. These input and output devices enable users to interact with the system in intuitive ways, supporting diverse functionalities from user control of applications to data visualization and multimedia output.

[0200] The systems and methods described herein are thus capable of deployment across a broad spectrum of computing environments, supporting applications from simple embedded systems to large-scale distributed computing networks. Each component and approach described herein contributes to the versatility and adaptability of the invention, making it suitable for a wide variety of practical implementations across industries and use cases.

[0201] The disclosed neural network implementations may be realized through various configurations of computer hardware, software, or a combination of both, depending on the requirements and constraints of the particular application. For instance, the neural networks may leverage specialized hardware, such as GPUs, tensor processing units (TPUs), FPGAs, and ASICs, which are designed to efficiently handle the high computational demands of training and deploying neural networks. These hardware components are particularly advantageous for accelerating matrix operations, which are central to neural network computations, and can significantly reduce the time needed for training large models and performing inference tasks.

[0202] Alternatively, neural networks can be implemented using traditional computer hardware, including CPUs, which are versatile and widely available. While CPUs are not optimized specifically for neural network computations, they can still handle smaller models and less computationally intensive tasks effectively. In cases where flexibility is essential, such as in general-purpose computing environments, implementing neural networks on CPUs allows for integration with other software systems without the need for specialized hardware.

[0203] On the software side, neural networks can be created using various programming languages and frameworks. High-level languages such as Python, Java, and C++ are commonly used for neural network development, particularly in conjunction with deep learning libraries and frameworks like TensorFlow, PyTorch, Keras, and Theano. These frameworks provide prebuilt functions, modules, and tools that simplify the process of designing, training, and deploying neural networks. They allow developers to define network architectures, optimize training parameters, and manage data flows with relative ease. For instance, TensorFlow and PyTorch offer extensive support for GPU and TPU integration, enabling seamless transitions between hardware and software environments.

[0204] Neural networks implemented in software may also vary based on the type of language and runtime environment used. For example, imperative languages such as Python and Java allow developers to create neural networks using clear, step-by-step procedural code, making the design process intuitive and manageable. Alternatively, functional languages like Lisp and Haskell may also be employed to build neural networks, particularly when focusing on functional aspects of data flow and transformation. Moreover, neural networks can be implemented in either compiled or interpreted languages, where compiled languages, such as C++ or Java, can offer improved execution speed, while interpreted languages like Python provide flexibility and ease of development.

[0205] In certain configurations, neural networks may be deployed in distributed computing environments, allowing for parallel processing across multiple processing units or even across different geographic locations. Distributed implementations can be achieved through cloud computing platforms or high-performance computing (HPC) systems, where workloads are split among numerous machines to improve efficiency and scalability. This is particularly useful for training large-scale models that require significant processing power and storage capacity. Distributed computing frameworks such as Apache Spark and Horovod can facilitate the parallelization of neural network computations, enablinglarge datasets and complex models to be processed in a fraction of the time that would be required on a single machine.

[0206] Neural network implementations may also employ hybrid configurations that combine both hardware and software elements. For example, the core neural network computations might be performed on dedicated hardware accelerators like GPUs or TPUs, while the overall system, including data preprocessing and postprocessing steps, can be managed by general-purpose software running on CPUs. This hybrid approach optimizes performance by leveraging the strengths of both hardware and software environments, ensuring efficient resource utilization across different components of the system.

[0207] Furthermore, it is understood that the neural networks described herein are not limited to any specific type of architecture. Various neural network architectures, including but not limited to convolutional neural networks (CNNs), recurrent neural networks (RNNs), long short-term memory (LSTM) networks, transformers, and generative adversarial networks (GANs), may be implemented depending on the task requirements. These architectures can be tailored to perform tasks such as image recognition, natural language processing, and predictive modelling, each benefiting from different configurations of hardware and software resources to optimize performance.

[0208] For secure and reliable deployment, neural networks may also incorporate mechanisms for data integrity, confidentiality, and robustness against adversarial attacks. Security protocols, such as data encryption and access control, may be applied to safeguard sensitive data processed by the neural network. Techniques like differential privacy and secure multiparty computation can be employed to protect data confidentiality during training and inference. Additionally, the implementation may include error-handling mechanisms and redundancy measures to ensure robust operation, even in environments where hardware failures or software bugs may occur.

[0209] Overall, the neural networks in this disclosure may be implemented as flexible, scalable systems that leverage combinations of hardware and software elements tailored to the needs of specific applications. This approach provides versatility, allowing the neural networks to be deployed in a wide range of environments, from dedicated hardware systems to virtualized cloud platforms, thereby supporting a diverse set of use cases and performance requirements.

[0210] In this disclosure, unless the context explicitly requires otherwise, the term “comprise” and its variations, such as “comprises” and “comprising,” are intended to be interpreted in an inclusive manner. This means that the presence of specified features or elements does not exclude the possibility of additional features, elements, or steps being included in various embodiments.

[0211] Any reference to prior art publications within this disclosure should not be taken as an acknowledgment or admission that these publications form part of the common general knowledge in the relevant field, whether in any particular jurisdiction or globally.

[0212] The examples provided in the above description serve to illustrate specific embodiments and convey certain features and principles. However, those skilled in the art will recognize that individual features, elements, and functionalities within the disclosed embodiments may be adapted, modified, or combined in numerous ways without departing from the core spirit or intended scope of the described subject matter. Therefore, the foregoing description is meant to be illustrative rather than limiting, with the scope being defined by the appended claims, which are intended to encompass all variations and modifications within the broadest interpretation permitted by applicable law.

Claims

CLAIMS1. An interactive installation, comprising:a physical engagement apparatus configured to support physical activity by a user;at least one motion sensor configured to detect an engagement of the user with the physical engagement apparatus and to generate at least one motion signal indicative of the engagement; andan electronic device configured to render an audio arrangement, the rendering being modulated by an input signal generated based at least in part on the motion signal.

2. The installation of claim 1 , wherein the motion signal is based on a level of kinetic energy generated by the engagement of the user with the physical engagement level.

3. The installation of claim 1 or 2, wherein the input signal is generated based on a momentum level computed by aggregating succesive motion signals generated by the motion sensor over a sliding time window.

4. The installation of any one of claims 1 to 3, wherein the physical engagement apparatus is a swing apparatus, and wherein the momentum level is computed based on at least one of: a magnitude of angular displacement over the sliding time window, a rate of rotation around at least one axis, a level of change in momentum between a present time and a previous time, and changes in rotation direction.

5. The installation of any one of claims 1 to 3, wherein the physical engagement apparatus is a stationary bicycle, and wherein the momentum level is computed based on at least one of: a rotation rate of a crankset and / or pedal assembly, a cadence of user pedalling over the sliding time window, a magnitude ofangular displacement of the crankset over the sliding time window, and a level of change in rotation speed between a present time and a previous time.

6. The installation of any one of claims 1 to 5, further comprising:at least one additional physical engagement apparatus configured to support additional physical activity by at least one additional user; andat least one additional sensor configured to detect an additional engagement of the additional user with the additional physical engagement apparatus and to generate at least one additional motion signal indicative thereof,wherein the input signal is generated based at least in part on the additional motion signal.

7. The installation of claim 6, wherein the audio arrangement comprises a plurality of tracks, and wherein a rendering of each track is modulated by an input signal generated based at least in part on the motion signal associated with the user or one of the additional motion signal associated with one or the at least one additional user.

8. The installation of any one of claims 1 to 7, wherein the motion sensor comprises at least one apparatus-mounted sensor secured to the physical engagement apparatus.

9. The installation of claim 8, wherein the apparatus-mounted sensor comprises an infrared sensor configured to detect the engagement of the user with the physical engagement apparatus.

10. The installation of claim 8 or 9, wherein the apparatus-mounted sensor comprises an imaging sensor configured to acquire images of the user and infer from the images the engagement of the user with the physical engagement apparatus.

11. The installation of any one of claims 1 to 10, wherein the motion sensor comprises at least one device-based sensor installed in a handheld device of the user configured to detect movements of the user and infer from the movements the engagement of the user with the physical engagement apparatus.

12. The installation of claim 11, wherein the electronic device is the handheld device.

13. The installation of claim 11 or 12, wherein the device-based sensor comprises at least one of: an inertial measurement unit, a gyroscope, an accelerometer, and a magnetometer.

14. The installation of any one of claims 1 to 13, further comprising a computer- readable code displayed on or near the physical engagement apparatus, wherein the electronic device is configured to read the computer-readable code and obtain the audio arrangement based at least in part on the computer- readable code.

15. The installation of any one of claims 1 to 14, wherein the electronic device further comprise at least one geolocation sensor configured to determine a geographic location, and the electronic device is configured to obtain the audio arrangement based at least in part on the geographic location.

16. A method of operating an interactive installation, the method comprising:detecting an engagement of a user with a physical engagement apparatus;generating at least one motion signal indicative of the engagement; andrendering an audio arrangement, the rendering being modulated by an input signal generated based at least in part on the motion signal.

17. The method of claim 16, wherein the motion signal is based on a level of kinetic energy generated by the engagement of the user with the physical engagement level.

18. The method of claim 16 or 17, wherein the input signal is generated based on a momentum level, further comprising computing the momentum level by aggregating succesive motion signals generated over a sliding time window.

19. The method of any one of claims 16 to 18, wherein the physical engagement apparatus is a swing apparatus, and wherein the momentum level is computed based on at least one of: a magnitude of angular displacement over the sliding time window, a rate of rotation around at least one axis, a level of change in momentum between a present time and a previous time, and changes in rotation direction.

20. The method of any one of claims 16 to 19, wherein the physical engagement apparatus is a stationary bicycle, and wherein the momentum level is computed based on at least one of: a rotation rate of a crankset and / or pedal assembly, a cadence of user pedalling over the sliding time window, a magnitude of angular displacement of the crankset over the sliding time window, and a level of change in rotation speed between a present time and a previous time.

21. The method of any one of claims 16 to 19, further comprising:detecting at least one additional engagement of at least one additional user with at least one additional physical engagement apparatus; andgenerating at least one additional motion signal indicative of the additional engagement,wherein the input signal is generated based at least in part on the additional motion signal.

22. The method of claim 21, wherein the audio arrangement comprises a plurality of tracks, and wherein rendering the audio arrangement comprises renderingeach track, the rendering of each track being modulated by an input signal generated based at least in part on the motion signal associated with the user or one of the additional motion signal associated with one or the at least one additional user.

23. The method of any one of claims 16 to 22, wherein detecting the engagement and / or generating the motion signal is at least in part performed by at least one apparatus-mounted sensor secured to the physical engagement apparatus.

24. The method of claim 23, wherein the apparatus-mounted sensor comprises an infrared sensor configured to detect the engagement of the user with the physical engagement apparatus.

25. The method of claim 23 or 24, wherein the apparatus-mounted sensor comprises an imaging sensor configured to acquire images of the user and infer from the images the engagement of the user with the physical engagement apparatus.

26. The method of any one of claims 16 to 25, wherein detecting the engagement and / or generating the motion signal is at least in part performed by at least one device-based sensor installed in a handheld device of the user configured to detect movements of the user and infer from the movements the engagement of the user with the physical engagement apparatus.

27. The method of claim 26, wherein the device-based sensor comprises at least one of: an inertial measurement unit, a gyroscope, an accelerometer, and a magnetometer.

28. The method of any one of claims 16 to 27, further comprising reading by an electronic device the computer-readable code and obtaining by the electronic device the audio arrangement based at least in part on the computer-readable code.

29. The method of any one of claims 16 to 27, further comprising determining by an electronic device a geographic location and obtaining by the electronic device the audio arrangement based at least in part on the geographic location.

30. An angular motion-based controller implemented in a handheld device, comprising:at least one motion sensor configured to measure a magnitude of an angular movement generated by a user engaging with a physical engagement apparatus;a processor configured to compute a momentum level of the handheld device based at least in part on the magnitude of the angular movement,a memory comprising instructions corresponding to at least one application, the application being configured to use the momentum level as an input parameter.

31. The controller of claim 30, wherein the angular movement is causes by an individual holding the handheld device while using a swing.

32. The controller of claim 30 or 31, wherein the application is an audio playback application configured to render an audio recording, the rendering being modulated on the input parameter.

33. The controller of claim 32, wherein the processor is configured to receive at least one additional momentum level of at least one additional handheld device from at least one additional controller, wherein the at least one additional momentum level of the at least one additional handheld device is provided as an additional input parameter to the application.

34. The controller of claim 33, wherein the audio recording comprises a plurality of tracks, wherein a rendeing of each one of the tracks is modulated based at least on the input parameter associated with the handheld device or the additional input parameter associated with the additional handheld device.

35. A method of operating an angular motion-based controller, the method comprising:measuring a magnitude of an angular movement generated by a user engaging with a physical engagement apparatus;computing a momentum level of the handheld device based at least in part on the magnitude of the angular movement,providing the momentum level as an input parameter of at least one application.

36. The method of claim 35, wherein the angular movement is caused by an individual holding the handheld device while using a swing.

37. The method of claim 35 or 36, further comprising rendering an audio recording by the application, the rendering being modulated on the input parameter.

38. The method of claim 37, further comprising:computing at least one additional momentum level associated with at least one additional handheld device from at least one additional controller; andproviding the at least one additional momentum level of the at least one additional handheld device as an additional input parameter to the application.

39. The method of claim 38, wherein the audio recording comprises a plurality of tracks, further comprising rendeing of each one of the tracks, the rendeing of each one of the tracks being modulated based at least on the input parameter associated with the handheld device or the additional input parameter associated with the additional handheld device.

40. A metacontroller system, comprising:an entertainment device configured to render a virtual environment based at least in part on control signals;a controller comprising a plurality of controls and configured to generate the control signals based on user interactions with the controls, the controls comprising a directional control and an action control;a physical engagement apparatus comprising at least one motion sensor configured to generate a motion signal based on detecting movement indicative of an engagement of the user with the physical engagement apparatus; anda metacontroller configured to receive the control signals from the controller and the motion signal from the exercise apparatus, and to forward at least a portion of the control signals to the electronic entertainment device based on the movement signal.

41. The system of claim 40, wherein the metacontroller is configured to forward the control signals only when the movement signal indicates detected movement.

42. The system of claim 40, wherein the metacontroller is configured to forward a first portion of the control signals corresponding to the action control at all times, and to forward a second portion of the control signals corresponding to the directional control only when the movement signal indicates detected movement.

43. The system of claim 40, wherein the metacontroller is configured to store a duration of the movement signal indicating detected movement and to forward the control signals for an interval of time based on the duration.

44. A method for controlling an electronic entertainment device based on a controller and a physical engagement apparatus, the method comprising:generating control signals based on actuation of the controller;detecting motion indicative of engagement of a user with the physical engagement apparatus;forwarding at least a portion of the control signals to the electronic entertainment device based on the motion detection; andrendering by the electronic entertainment device a virtual environment based on the forwarded portion of the control signals.

45. The method of claim 44, wherein the forwarding comprises forwarding the control signals only when the motion is detected.

46. The method of claim 44, wherein the forwarding comprises:forwarding a first portion of the control signals corresponding to the action control at all times; andforwarding a second portion of the control signals corresponding to the directional control only when the movement signal indicates detected movement.

47. The method of claim 40, further comprising storing a duration of the motion being detected, wherein forwarding comprises forwarding the control signals for an interval of time based on the duration.

48. A metacontroller configured to:receive control signals from a controller comprising a plurality of controls and configured to generate the control signals based on user interactions with the controls, the controls comprising a directional control and an action control, and motion signal from a physical engagement apparatus comprising at least one sensor configured to generate the motion signal based on detecting engagement of a user with the physical engagement apparatus; andforward at least a portion of the control signals to an entertainment device configured to render a virtual environment based at least in part on control signals based on the movement signal.

49. A method of operating a metacontroller, the method comprising:receiving control signals from a controller comprising a plurality of controls and configured to generate the control signals based on user interactions with the controls, the controls comprising a directional control and an action control, and motion signal from a physical engagement apparatus comprising at least one sensor configured to generate the motion signal based on detecting engagement of a user with the physical engagement apparatus; andforwarding at least a portion of the control signals to an entertainment device configured to render a virtual environment based at least in part on control signals based on the movement signal.

50. An interactive installation, comprising:a display;an electronic entertainment device configured to render an avatar in a virtual environment to the display based on first and second control signals;a stationary bicycle comprising a crankset and a motion sensor operably coupled to the electronic entertainment device and configured to function as a first controller of the electronic entertainment device by generating the first control signals in response to a user actuating the crankset;a tilt-based controller operably coupled to the electronic entertainment device and configured to function as a second controller of the electronic entertainment device, the tilt-based controller comprising:an imaging sensor configured to acquire an image of the user,a memory comprising a machine learning model trained to accept the image as input and predict coordinates of at least two primary anatomical landmarks, anda processor configured to:generate the coordinates of the at least two primary anatomical landmarks by executing the machine learning model,generate the coordinates of two secondary anatomical landmarks based on the at least two primary anatomical landmarks,compute an angle formed by a straight line connecting the two secondary anatomical landmarks and a reference axis, andgenerate the second control signal based on the angle.

51. An electronic entertainment system, comprising:a display;an electronic entertainment device configured to render an avatar in a virtual environment to the display based on control signals;a controller comprising a plurality of controls and configured to generate the control signals based on user interactions with the controls, the controls comprising a directional control and an action control;a stationary bicycle comprising a crankset and a motion sensor configured to generate a movement signal in response to a user actuating the crankset; anda metacontroller operably coupled to the electronic entertainment device, the controller and the motion sensor, the metacontroller comprising a processor configured to:receive the control signals from the controller,forward the control signals associated with the action control to the electronic entertainment device, andin response to receiving the movement signal, forward the control signals associated with the directional control to the electronic entertainment device.

52. A tilt-based controller system, comprising:an exercise apparatus;a display;an electronic entertainment device configured to render to the display a virtual environment based at least in part on control signals;an imaging sensor configured to acquire an image of an individual using the exercise apparatus; anda processor configured to:run a machine learning model trained to accept the image as input and predict coordinates of at least two primary anatomical landmarks, andcompute an angle formed by: a straight line connecting two secondary anatomical landmarks based on the coordinates of the at least two primary anatomical landmarks, and a reference axis,wherein the angle is provided as part of the control signals to the electronic entertainment device.

53. The system of claim 52, wherein the angle is the result of the individual leaning to a side.

54. The system of claim 52 or 53, wherein the virtual environment comprises an avatar and wherein the entertainment device is configured to render a lateral movement of the avatar based on the angle.

55. A method for controlling an electronic entertainment device based on a pose of an individual, the method comprising:capturing an image of the individual;predicting coordinates of at least two anatomical landmarks;computing the pose of the individual based on the coordinates of the at least two primary anatomical landmarks; andrendering by the electronic entertainment device a virtual environment based on the pose of the individual.

56. The method of claim 55, wherein the pose indicates whether the individual is leaning to a side.

57. A tilt-based controller, comprising:an imaging sensor configured to acquire an image of an individual using an exercise apparatus; anda processor configured to:run a machine learning model trained to accept the image as input and predict coordinates of at least two primary anatomical landmarks, andcompute an angle formed by: a straight line connecting two secondary anatomical landmarks based on the coordinates of the at least two primary anatomical landmarks, and a reference axis,wherein the angle is provided as part of the control signals to an electronic entertainment device configured to render to a display a virtual environment based at least in part on control signals.

Images