Strength training device form training system

Abstract

An apparatus, method, and system for form training with a strength training device having at least one sensor. The form training includes obtaining, from the at least one sensor, sensor data including a person performing an exercise action with the strength training device. The form training includes an exercise form with point(s) tracking the exercise action performed by the person, each point representing a location on the person. The form training includes generating, by a model and for the exercise form, an evaluation indicating a deviation of the exercise form from a target form for performing the exercise action. The form training includes determining, based on the evaluation for the exercise form, a form adjustment of the person for performing the exercise. The form training includes providing, by a display device, data indicative of the form adjustment for output on the display device.

Description

STRENGTH TRAINING DEVICE FORM TRAINING SYSTEMCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority to U. S. Provisional Application No.63 / 718,337 filed on November 8, 2024, the contents of which are hereby incorporated by reference.BACKGROUND

[0002] This disclosure relates generally to sports training and form using exercise equipment.

[0003] An athlete can perform a number of different exercises using a piece of exercise equipment. The athlete’s form in performing an exercise can include appropriate technique for the exercise, alignment of different parts of the body relative to each other and / or the piece of exercise equipment, among other aspects of training and sports. Performing an exercise with proper form is important to obtaining desired results from exercising, while also mitigating risks associated with exercise such as injury, chronic pain, muscle imbalance, etc.

[0004] Thus, there is a need for improving an athlete’s form in performing exercise.SUMMARY

[0005] This specification describes a method, system, and a non-transitory computer-readable storage medium for a strength training device form training system (also referred to as “a form training system” or “a system”) configured to apply computer vision techniques for training athletes performing exercises using a piece of exercise equipment, such as a sled. The disclosed techniques include obtaining, from sensors of the strength training device form training system, sensor data of a person performing an exercise action with a strength training device and determining an exercise form for the person. An exercise form can include biometric points tracking the exercise action performed by the person, each point from the one or more points representing a location on and / or relative to the person. The system generates, based on the exercise form, an evaluation indicating a deviation of the exercise form from an ideal form for performing the exercise action.

[0006] The strength training device form training system can be used with, for example, an American football training sled. An American football training sled (also referred to as a “football sled” or a “sled”) is a piece of exercise equipment that an athlete can use for Americanfootball training by performing full-body exercises using the sled. Examples of exercises with the sled can include pushing the sled, dragging the sled, lifting the sled, flipping the sled, etc.

[0007] However, the sled need not be limited to American football training, as the exercises can also be performed for strength training in other applications, such as recreational fitness, general fitness, first responder physical training, etc. These applications can demand training with a sled-type strength training device (among other types of strength training devices) to improve a user’ s strength (e.g., an ability to carry a load with a particular weight) and endurance (e.g., an ability to carry a load for an extended period of duration). For example, a backpacker can perform exercises using a strength training device for endurance and strength training, so that they can carry significant weight in backpacks over long periods time. As another example, first responders such as search and rescue personnel, firefighters, police officers, etc., can perform exercises using a strength training device for first responder training, due to training demands to carry heavy loads such as equipment, people, debris, etc.

[0008] An athlete can perform different types of exercises using the sled relative to a surface of an athlete’s training environment. For example, some complex exercises for American football training can include performing a “clean, drive, flip” (CDF) with the sled. A CDF is where the athlete performs a clean with the sled, e.g., by lifting one end of the sled using an overhand grip to lift the sled from the ground at an angle, then performs a drive with the sled, e.g., by pushing the sled along a ground surface while holding the sled at an angle, and then flips the sled, e.g., by pushing the end of the sled that is being held so that the sled falls over towards the direction the sled was being pushed towards. Other examples of complex exercises using a strength training device (such as a sled) can include dragging and pulling the strength training device, performing a clean and a press using the strength training device, as well as deadlifting the strength training device, among other examples described in this specification. These types of exercises (also referred to as a sequence of power moves) can include multiple phases, which can include a number of pulls, pushes, positioning and / or re-positioning (e.g., stance, grip, posture), recovery, etc. to perform the exercise.

[0009] By collecting and processing athlete data, e.g., images, videos, biometrics, audio, of an athlete performing an action, e.g., during a static or dynamic lifting, pushing, flipping action, or the like, the strength training device form training system can capture biomechanical information of the athlete. The system can identify athlete form deviation with respect to an ideal form and / or self-consi stent form, correlate the deviations with corrective drills and standards for exercises using the strength training device, and provide feedback to the athlete.This can allow for real-time feedback and corrective action for the athlete, summary and statistical information about the practice session, and accurate form tracking and correction.

[0010] The form training system can capture sensor data, e.g., video, via one or more sensors on or attached to a strength training device or can receive captured sensor data from the one or more sensors that are separate from the strength training device. For example, the computer vision system can receive video data captured by a mobile computing device of a user (e.g., a cellular phone or tablet with a camera). An application on the device can record or transmit video, or data from the video, to the strength training device form training system, or the application on the device itself may perform one or more types of computer-vision analysis.

[0011] The system can integrate mathematical representations of an exercise action, e.g., pulling a strength training device, flipping a strength training device, pushing a strength training device, and identify a subset of points of the biomechanical data extracted from the video data captured of the exercise action to represent the different biomechanical phases of the exercise action. The system can assess the exercise over multiple performed exercise actions, e.g., a sequence of exercises, to determine consistency of the athlete with respect to an ideal form and / or self-consistent form.

[0012] This can advantageously allow for more accurate, dynamic, and proactive strength training device training, such as football sled training, other types of sports training, general fitness, first responder physical training (e.g., for police officers, firefighters) than otherwise possible with other strength training device training systems that lack these features. A strength training device that does not have this type of computer vision analysis would be unable to observe an athlete’s form as they perform an exercise action with the strength training device. For example, an exercise action can include the athlete’s stance prior to interacting with the sled (e.g., an example strength training device), the athlete’s grip with the sled, e.g., a portion of the sled where the athlete’s grip and / or the type of grip applied to the sled, the athlete’s motions while they apply force to lift the sled up, the athlete’s posture while performing motions, and other related motions while the athlete pushes the sled. Thus, a strength training device without sensor-based computer vision analysis does not provide useful feedback that could overwise be determined and provided by the disclosed sensor-based strength training device form training system. Furthermore, the system can provide an analysis of the athlete’s form, recommendations for drills specific to a form of a given athlete, including recommendations indicating when an athlete is using inadequate form when performing the exercise. The disclosed technology allows for improvements from inadequate form by anathlete when performing an exercise, such as an incorrect stance, grip, pushing and / or pulling motions, flipping, cleans, and other types of combined moves.

[0013] In some implementations, the strength training device form training system including a sensing subsystem including a non-visible wavelength source, e.g., emitter, and a sensor, e.g., detector, configured to detect reflections of the source from the training environment surrounding the strength training device form training system. In such instances, the wavelengths emitted by the source can be, for example, millimeter wave (mmWave), LIDAR, pulsed coherent radar, microwave radar, or other non-visible wavelength bands of the electromagnetic spectrum. An operating wavelength band of the sensing subsystem can be selected in part using various factors, for example, accuracy, precision, sensing speed, resolution, and range of detection. The operating wavelength band can be selected, for example, based on a resolution requirement for classifying the different aspects of the exercise action.

[0014] Non-visible wavelength-based sensing subsystems can offer increased robustness to environmental factors over visible-based sensing, e.g., cameras. For example, non-visible based sensing can be robust to lighting conditions, weather interference (e.g., rain), contaminants such as dust or other particles, and the like. In some implementations, non-visible frequency-based sensing can be incorporated in addition to, or alternatively to, camera-based sensing in the strength training device form training system. Non-visible sensing data, e.g., point cloud data, can be used instead of or to enrich visible-wavelength camera-based imaging data. In some examples, non-visible wavelength sensing subsystems can increase privacy of users by limiting feature resolution of the collected sensor data, e.g., not including facial information.

[0015] In some examples, a mmWave-based sensing subsystem can provide high-precision, low latency object (e.g., athlete and strength training device) sensing capabilities. A mmWave-based sensing subsystem can provide added robustness, e.g., over alternative radar technologies, to environmental interference. In some examples, mmWave-based sensing can additionally include Doppler functionality, e.g., measuring speed / velocity of an athlete before, during, and after the exercise action.

[0016] For example, a mmWave-based sensing subsystem can offer increased robustness to variations in lighting conditions, e.g., low light, dust / particles, and weather conditions such as rain or fog (if in an outdoor environment). In cases where mmWave-based sensing is used, thecollected sensor data, e.g., raw point cloud data, can be processed to protect the privacy of the athletes captured in the sensor data.

[0017] The system can capture sensor data from two or more sensors of the player performing an action. The two or more sensors can be the same type of sensor, e.g., two or more cameras, two or more mmWave sensors, or the like. In some instances, the two or more sensors can be different types of sensors, where the system captures respective types of sensor data from the two or more different types of sensors of the player performing the action and merges the multiple streams of sensor data to generate an action profile for the action. Video data can be enriched with additional sensor data collected from one or more other types of sensors in addition to the video data captured using a camera. For example, mmWave radar data can be used to enrich video data to provide metadata for the action captured using the camera.

[0018] One general aspect of a form training system includes a strength training device; at least one sensor; and one or more processors coupled to one or more computer-readable storage media having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations includes: obtaining, from the at least one sensor, sensor data including a person performing an exercise action with the strength training device; determining, from the sensor data, an exercise form includes one or more points tracking the exercise action performed by the person, each point from the one or more points representing a location on the person; generating, by a model and for the exercise form, an evaluation indicating a deviation of the exercise form from a target form for performing the exercise action; determining, based on the evaluation for the exercise form, a form adjustment of the person for performing the exercise; and providing, by a display device, data indicative of the form adjustment for output on the display device.

[0019] Implementations can include one or more of the following features.

[0020] The strength training device is a football training sled. The exercise includes one or more of (i) flipping, (ii) lifting, or (iii) pushing, the strength training device. The exercise includes one or more motions performed by the person using the strength training device.

[0021] The sensor data is captured by the at least one sensor prior to the exercise being performed. The sensor data is captured by the at least one sensor while the exercise is performed by the person. The sensor data is captured by the at least one sensor after the exercise is performed by the person.

[0022] The model is configured to apply one or more of (i) machine learning techniques, (ii) statistical techniques, or (iii) a physical -based simulation, to generate the evaluation. Theevaluation includes one or more of (i) a label, or (ii) a score, representing a deviation of the exercise form from the target form.

[0023] The operations can include: determining, based on the evaluation for the exercise form, a form adjustment of the person for performing the exercise; and providing data indicative of the form adjustment for output. Providing data indicative of the form adjustment includes providing the data for display on a user interface of a device communicatively coupled to the one or more processors, wherein the data causes the device to configure the user interface to display one or more user interface elements to display the form adjustment.

[0024] The at least one sensor is mounted onto the strength training device. The at least one sensor is located at a position remote from the strength training device. The at least one sensor is co-located with the one or more processors. The at least one sensor includes is one or more of (i) an image sensor, (ii) a proximity sensor, (iii) an infrared sensor, or (iv) a radar sensor.

[0025] The plurality of points track respective locations of biometric data points corresponding to the person in the sensor data and tracking the exercise action; identifying, from the action data, two or more phases of the exercise action, wherein each of the two or more phases of the exercise action include a subset of the plurality of points tracking a timing and movement of the phase of the exercise action; and generating, from the action data, an exercise profile for the exercise action includes the two or more phases.

[0026] The evaluating includes: providing, for each of the two or more phases and to a corresponding trained machine learning model of a plurality of trained machine learning models, a subset of the plurality of points tracking the timing and movement of the phase of the exercise action; and obtaining, for each of the two or more phases of the exercise action, phase metrics for the phase.

[0027] The plurality of trained machine learning models are each trained on a respective, different data set corresponding to a phase of the two or more phases of a exercise profile for the exercise action.

[0028] The operations further include: evaluating, from the phase metrics of each phase of the two or more phases of the exercise profile, the exercise profile against a target exercise profile; determining, from the evaluation, a deviation of the exercise profile from the target exercise profile for a phase of the two or more phases is larger than a threshold deviation; and providing, to the person, a training alert. Providing the training alert further includes: determining, for the phase having a deviation larger than the threshold deviation, a training regimen specific to the phase and different than a training regimen for each other phase of the two or more phases; andproviding, to the person, the training alert. Determining a deviation of the exercise profile from the ideal exercise profile for the phase includes determining, a type of deviation of a plurality of types of deviations for the phase, and wherein determining the training regimen specific to the phase includes determining the training regimen from a plurality of training regimens that is specific to the type of deviation of the plurality of types of deviations for the phase. Providing the training alert includes: providing, for presentation in a user interface, a visual representation indicating the phase of the two or more phases for which the deviation of the exercise profile from the ideal exercise profile is larger than the threshold deviation; and providing, for presentation in the user interface, a selectable control operable to initiate the training regimen specific to the phase.

[0029] The action data includes the plurality of points tracking the exercise action further includes tracking respective locations of an object captured by in the sensor data, and wherein tracking the respective locations of the object includes tracking the object relative to the biometric data points corresponding to the person in the sensor data. Each phase of the two or more phases is defined by coordinate points and vectors for the points through a duration of the phase of the exercise action. At least one phase of the two or more phases overlaps in timing with at least one other phase of the two or more phases during the exercise action. Obtaining the sensor data includes obtaining the sensor data including the person performing the exercise action two or more times, and wherein generating an exercise profile includes generating a respective exercise profile for each of the two or more performed exercise actions by the person. Obtaining the sensor data further includes: identifying, for each of the respective exercise profiles, a corresponding reference point of the exercise profile; and aligning the exercise profiles for the two or more performed exercise actions using the respective reference points of the exercise profiles.

[0030] The operations further include: generating, from each of the two or more performed exercise actions, an average exercise profile for the exercise action for the person; and evaluating, from the average exercise profile against a target exercise profile; determining, from the evaluation, a deviation of the average exercise profile from the target exercise profile for a phase of the exercise action is larger than a threshold deviation; and providing, to the person, a training alert. The at least one sensor includes an image sensor, and wherein the sensor data includes video data including a plurality of frames including the person performing the exercise action. Obtaining, from the image sensor, the video data includes the plurality of frames including the person performing the exercise action includes: providing, forpresentation in a user interface, a first visual indication of a first position to arrange the image sensor with respect to the person performing the exercise action to capture the video data; receiving, from the image sensor, the video data; determining, from the video data, that a threshold video data capturing the exercise action from the first position is met; and providing, for presentation in the user interface, a second visual indication of a completion of capture of the video data from the first position. Obtaining, from the image sensor, the video data includes the plurality of frames including the person performing the exercise action further includes: providing, for presentation in a user interface, a third visual indication of a second position to arrange the image sensor with respect to the person performing the exercise action to capture the video data, wherein the second position is at least one of (i) a different position, or (ii) at a different angle, with respective to the form training system from the first position; receiving, from the image sensor, the video data; determining, from the video data, that a threshold video data capturing the exercise action from the second position is met; and providing, for presentation in the user interface, a fourth visual indication of a completion of capture of the video data from the second position. The first position is a side view of the person performing the exercise action, and wherein the second position is a face-on view of the person performing the exercise action. Determining, from the video data, that the threshold video data capturing the exercise action from the first position and the second position is met includes: determining that a threshold number of exercise actions are captured in the plurality of frames of the video data from the respective first position and the second position. The threshold number of exercise actions is different between the first position and the second position. Obtaining, from the at least one sensor, the sensor data including the person performing the exercise action further includes: providing, to a trained machine learning model, the sensor data capturing the exercise action from the first position; and obtaining, from the trained machine learning model, extrapolated synthetic sensor data representing the exercise action from a second, virtual position.

[0031] The operations further include: obtaining, from the strength training device and for the exercise action, exercise action completion feedback; and generating, from an exercise profile and the exercise action completion feedback, an enriched exercise profile for the exercise action. The at least one sensor is configured to capture sensor data in a non-visible electromagnetic spectrum frequency band. The at least one sensor is configured to capture sensor data in one or more of a non-visible electromagnetic spectrum frequency band, radio frequency bands, microwave frequency bands, acoustic frequency bands, ultrasonic frequencybands, and infrared frequency bands. The sensor data includes point cloud data. The at least one sensor is configured to capture sensor data in an ultrasonic frequency band. The at least one sensor includes a narrow-beam ultrasonic sensor. Implementations of the described techniques can include hardware, a method or process, or computer software on a computer-accessible medium.

[0032] One general aspect includes a form training method for a strength training device. The form training method also includes identifying, from the action data, one or more phases of the exercise action, wherein each of the one or more phases of the exercise action include a proper subset of the plurality of points tracking a timing and movement of the phase of the exercise action; and generating, from the action data, a movement profile for the exercise action includes the one or more phases.

[0033] One general aspect includes a form training system that also includes a strength training device; at least one sensor; and one or more processors coupled to one or more computer-readable storage media having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations including: obtaining, from the at least one sensor, sensor data including a person performing an exercise action with the strength training device; determining, from the sensor data, an exercise form includes one or more points tracking the exercise action performed by the person, each point from the one or more points representing a location on the person; and generating, based on the exercise form, a movement profile for the exercise action.

[0034] Implementations can include one or more of the following features. The strength training device is a football training sled. The exercise includes one or more of (i) flipping, (ii) lifting, or (iii) pushing, the strength training device. The exercise includes one or more motions performed by the person using the strength training device. The sensor data is captured by the at least one sensor prior to the exercise being performed. The sensor data is captured by the at least one sensor while the exercise is performed by the person. The sensor data is captured by the at least one sensor after the exercise is performed by the person. A model is configured to apply one or more of (i) machine learning techniques, (ii) statistical techniques, or (iii) a physical -based simulation, to generate an evaluation of the exercise form. An evaluation of the exercise form includes one or more of (i) a label, or (ii) a score, representing a deviation of the exercise form from a target exercise form.

[0035] The operations include: determining, based on an evaluation for the exercise form, a form adjustment of the person for performing the exercise; and providing data indicative of theform adjustment for output. Providing data indicative of the form adjustment includes providing the data for display on a user interface of a device communicatively coupled to the one or more processors, wherein the data causes the device to configure the user interface to display one or more user interface elements to display the form adjustment. The at least one sensor is mounted onto the strength training device. The at least one sensor is located at a position remote from the strength training device. The at least one sensor is co-located with the one or more processors. The at least one sensor includes is one or more of (i) an image sensor, (ii) a proximity sensor, (iii) an infrared sensor, or (iv) a radar sensor. The plurality of points track respective locations of biometric data points corresponding to the person in the sensor data and tracking the exercise action; identifying, from the action data, one or more phases of the exercise action, wherein each of the one or more phases of the exercise action include a subset of the plurality of points tracking a timing and a movement of the phase of the exercise action; and generating, from the action data, an exercise profile for the exercise action includes the one or more phases. The evaluating includes: providing, for each of the one or more phases of the exercise action and to a corresponding trained machine learning model of a plurality of trained machine learning models, the subset of the plurality of points tracking the timing and the movement of the phase of the exercise action; and obtaining, for each of the one or more phases of the exercise action, phase metrics for the phase. The plurality of trained machine learning models are each trained on a respective, different data set corresponding to a phase of the one or more phases of a exercise profile for the exercise action.

[0036] The operations further include: evaluating, from the phase metrics of each phase of the one or more phases of the exercise profile, the exercise profile against a target exercise profile; determining, from the evaluation, a deviation of the exercise profile from the target exercise profile for a phase of the exercise action is larger than a threshold deviation; and providing, to the person, a training alert. Providing the training alert further includes: determining, for the phase having a deviation larger than the threshold deviation, a training regimen specific to the phase and different than a training regimen for each other phase of the one or more phases; and providing, to the person, the training alert. Determining the deviation of the exercise profile from the ideal exercise profile for the phase includes determining, a type of deviation of a plurality of types of deviations for the phase, and wherein determining the training regimen specific to the phase includes determining the training regimen of a plurality of training regimens that is specific to the type of deviation of the plurality of types of deviations for the phase. Providing the training alert includes: providing, for presentation in a user interface, avisual representation indicating the phase of the one or more phases for which the deviation of the exercise profile from the ideal exercise profile is larger than the threshold deviation; and providing, for presentation in the user interface, a selectable control operable to initiate the training regimen specific to the phase. The action data includes the plurality of points tracking the exercise action further includes tracking respective locations of an object captured by in the sensor data, and wherein tracking the respective locations of the object includes tracking the object relative to the biometric data points corresponding to the person in the sensor data. Each phase of the one or more phases is defined by coordinate points and vectors for the points through a duration of the phase of the exercise action. The action data includes two or more phases of the exercise action and at least one phase of the two or more phases overlaps in timing with at least one other phase of the two or more phases during the exercise action. Obtaining the sensor data includes obtaining the sensor data including the person performing the exercise action two or more times, and wherein generating an exercise profile includes generating a respective exercise profile for each of the two or more performed exercise actions by the person. Obtaining the sensor data further includes: identifying, for each of the respective exercise profiles, a corresponding reference point of the exercise profile; and aligning the exercise profiles for the two or more performed exercise actions using the respective reference points of the exercise profiles.

[0037] The operations further include: generating, from each of the two or more performed exercise actions, an average exercise profile for the exercise action for the person; and evaluating, from the average exercise profile against a target exercise profile; determining, from the evaluation, a deviation of the average exercise profile from the target exercise profile for a phase of the exercise action is larger than a threshold deviation; and providing, to the person, a training alert. The at least one sensor includes an image sensor, and wherein the sensor data includes video data including a plurality of frames including the person performing the exercise action. Obtaining, from the image sensor, the video data includes the plurality of frames including the person performing the exercise action includes: providing, for presentation in a user interface, a first visual indication of a first position to arrange the image sensor with respect to the person performing the exercise action to capture the video data; receiving, from the image sensor, the video data; determining, from the video data, that a threshold video data capturing the exercise action from the first position is met; and providing, for presentation in the user interface, a second visual indication of a completion of capture of the video data from the first position. Obtaining, from the image sensor, the video data includesthe plurality of frames including the person performing the exercise action further includes: providing, for presentation in the user interface, a third visual indication of a second position to arrange the image sensor with respect to the person performing the exercise action to capture the video data, wherein the second position is at least one of (i) a different position, or (ii) at a different angle, with respective to the form training system from the first position; receiving, from the image sensor, the video data; determining, from the video data, that a threshold video data capturing the exercise action from the second position is met; and providing, for presentation in the user interface, a fourth visual indication of a completion of capture of the video data from the second position. The first position is a side view of the person performing the exercise action, and wherein the second position is a face-on view of the person performing the exercise action. Determining, from the video data, that the threshold video data capturing the exercise action from the first position and the second position is met includes: determining that a threshold number of exercise actions are captured in the plurality of frames of the video data from the respective first position and the second position. The threshold number of exercise actions is different between the first position and the second position. Obtaining, from the at least one sensor, the sensor data including the person performing the exercise action further includes: providing, to a trained machine learning model, the sensor data capturing the exercise action from a first position; and obtaining, from the trained machine learning model, extrapolated synthetic sensor data representing the exercise action from a second, virtual position.

[0038] The operations further include: obtaining, from the strength training device and for the exercise action, exercise action completion feedback; and generating, from an exercise profile and the exercise action completion feedback, an enriched exercise profile for the exercise action. The at least one sensor is configured to capture sensor data in a non-visible electromagnetic spectrum frequency band. The at least one sensor is configured to capture sensor data in one of radio frequency bands, microwave frequency bands, acoustic frequency bands, and infrared frequency bands. The sensor data includes point cloud data. The at least one sensor is configured to capture sensor data in an ultrasonic frequency band. The at least one sensor includes a narrow-beam ultrasonic sensor. Implementations of the described techniques can include hardware, a method or process, or computer software on a computer-accessible medium.

[0039] One general aspect includes of a form training system also includes a strength training device; at least one sensor; and one or more processors coupled to one or more computer-readable storage media having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations includes: obtaining, from the at least one sensor, sensor data including a person performing an exercise action with the strength training device; determining, from the sensor data, an exercise form includes one or more points tracking the exercise action performed by the person, each point from the one or more points representing a location on the person; and generating, by a model and for the exercise form, an evaluation indicating a deviation of the exercise form from a target form for performing the exercise action.

[0040] Implementations can include one or more of the following features. The system wherein the at least one sensor is one of (i) mounted onto the strength training device, (ii) located at a position remote from the strength training device, and (iii) co-located with the one or more processors. The sensor data is captured by the at least one sensor during a time period includes at least one of (i) prior to the exercise action being performed by the person, (ii) while the exercise action is performed by the person, and (iii) after the exercise action is performed by the person. The sensor data is captured by the at least one sensor prior to the exercise being performed. The sensor data is captured by the at least one sensor while the exercise is performed by the person. The sensor data is captured by the at least one sensor after the exercise is performed by the person. The exercise action includes one or more of (i) flipping, (ii) lifting, or (iii) pushing, the strength training device. The exercise includes one or more motions performed by the person using the strength training device. Implementations of the described techniques can include hardware, a method or process, or computer software on a computer-accessible medium.

[0041] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0042] FIG. 1 A is a diagram of an example sensor-based strength training device form training system.

[0043] FIG. IB is a diagram of the example exercise actions using a strength training device of the strength training device form training system.

[0044] FIG. 2 is a block diagram of an example operating environment of a strength training device form training system.

[0045] FIG. 3 A is a flow diagram of an example process of the sensor-based strength training device form training system.

[0046] FIG. 3B is a flow diagram of another example process of the sensor-based strength training device form training system.

[0047] FIG. 4 is a flow diagram of another example process of the strength training device form training system.

[0048] FIG. 5 is a flow diagram of another example process of the strength training device form training system.

[0049] FIG. 6 is a flow diagram of another example process of the strength training device form training system.

[0050] FIG. 7 is a schematic of an example of a biomechanical pose detection schema.

[0051] FIG. 8 is an example of a user interface of the sensor-based strength training device form training system.

[0052] FIG. 9 is an example of a user interface of the sensor-based strength training device form training system.

[0053] FIG. 10 is an example computer device for the sensor-based strength training device form training system.

[0054] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0055] An inadequate (e.g., improper, incorrect) form can result in a person being unable to successfully perform a simple (e.g., a single movement) or complex motion (e.g., a combination of movements) to perform an action. For example, an athlete performing an exercise (e.g., an exercise action) using inadequate form may be unable to push or pull a piece of training equipment such as a sled, and may also be unable to lift the sled and / or push the sled after lifting to flip the sled. For general fitness, a person using inadequate form may be unable to lift a strength training device with the desired weight but may also risk injury by using improper form. As another example, a first responder (e.g., police, firefighter) may perform physical training that include complex actions using strength training devices, andinadequate form in these scenarios can also result in injury or poor training, e.g., for rescue operations, team exercises.

[0056] In some cases, inadequate form can result in the athlete performing the exercise action without engaging a target muscle or muscle group that the exercise action is designed to train, resulting in an exercise action that does not provide the desired training and exercise to the target muscle or muscle groups. In some cases, an inadequate or incorrect form can result in a risk of injury to the athlete or damage to the strength training device. As another example, an inadequate form of an exercise includes performing the exercise without performing the full range of motion associated with the exercise. While the exercise may be fully performed by the athlete without completing the full range of motion, the form of the athlete performing the exercise can be considered improper because the benefits associated with performing the full range of motion are not fully realized. By analyzing and providing corrections (e.g., in the form of adjustments) to improper form, the disclosed technology reduces injury risk and damage to athletic equipment, improving safety for the athlete and extending the lifespan of exercise equipment, respectfully.

[0057] The disclosed technology allows for personalized, sensor-based recommendations for correcting form. While two athletes may both be able to successfully lift and push the strength training device, e.g., to flip the strength training device, one athlete may be adopting an improper form that may cause injury, e.g., locking their knees when performing a lift portion to flip the strength training device. The second athlete may be performing the lift portion of the flipping exercise with correct lifting technique, but may be pushing the strength training device with poor form, e.g., with poor posture, front foot placement, poor grip and / or hand position, and a lack of alignment between the athlete’s body and the direction of the push. With this information, the two athletes can be provided with recommendations for different drills, such as drills to improve posture, foot placement, body alignment, hand / grip position, and engaging the correct muscle groups.Strength Training Device Form Training System

[0058] FIG. 1 A shows a training environment 100 that includes a strength training device form training system 101 configured to capture sensor data of an athlete 102 performing an exercise action 106 with a strength training device 104. The strength training device form training system 101 can be referred to as “system 101.” The strength training device 104 can be, forexample, an American football training sled. The strength training device 104 can optionally include additional components, such as weighted plates. The system 101 is depicted in FIG.1A as having a user device 130-1 (also referred to as a computing device 130-1) that further includes a sensor 120-1, a model 108, and a form training engine 110. Although FIG. 1A illustrates a single computing device 130-1, any number of computing devices 130-N can be employed by the system 101. The model 108 can be referred to as a pose information model 108 or a form training model 108.

[0059] While described herein with respect to a system 101 for performing exercises with a strength training device such as a sled, it should be understood that aspects of system 101 can be applied to other types of exercise equipment as well. For instance, system 101 can be used for training with equipment such as medicine balls, squat racks, free weights, pommel horse, etc. for training purposes in other sports. The system 101 can be considered, in some examples, as a sports form training machine or an exercise form training machine.

[0060] The sensor 120-1 is shown as co-located with the computing device 130-1, though the sensors of the system 101 can be remote from the computing device 130-1, such as sensor 120-2. A sensor from sensors 120 can also be referred to as “sensor 120”, but the system 101 can include any number of sensors 120-1 through 120-N (collectively referred to as “sensors 120”). The sensors 120 are configured to capture sensor data, such as sensor measurements, detections, etc. of the athlete 102 prior to, during, and / or after completing, the exercise action 106 with the strength training device 104. The system 101 also includes a sensing subsystem 124 that includes a sensor 120-N and a source 122, in which source 122 is configured to transmit a signal in the training environment 100. The sensor 120-N is configured to detect signals and determine a receive signal including measurements resulting from the transmit signal of source 122. In some cases, a sensor can be mounted onto and / or be a part of the strength training device 104, such as sensor 120-3 depicted in FIG. 1 A.

[0061] The computing device 130-1 can be any computer device, system, or platform. The computing device 130-1 is communicatively coupled to sensors 120, e.g., sensor 120-1 through 120-N, by communication network 112. The system 101 can utilize a communication network 112 that utilizes one or more communication device(s) to communicate with external devices via one or more wired or wireless communication networks, or both. Communication device(s) can include any one or more communication devices, such as network interface cards (e.g., Ethernet cards), optical transceivers, radio frequency transceivers, Bluetooth transceivers, 3G or 4G transceivers, and WiFi radio computing devices.

[0062] The computing device 130-1 includes a form training engine 110 that is coupled to the sensor 120-1 and the model 108. The form training engine 110 can receive sensor data from sensor 120-1 and use the sensor data as an input to model 108. The model 108 can be configured to apply or perform a number of machine learning techniques, statistical techniques, physicsbased simulations, to generate a model output indicating an evaluation of the form of athlete 102 for performing an exercise action 106 with strength training device 104. In some cases, the model output from model 108 can also include data indicating a form adjustment for the athlete to improve the form of the exercise action 106 being perform. In some cases, the model output can include graphical interface data that configures a display of a device (e.g., a user device 130-1) to display the form adjustment, but the model output can also be an alert provided to the athlete 102.

[0063] The form training engine 110 can also be configured to train the model 108 using sensor data from sensor 120-1. The form training engine 110 can also be configured to train the model using sensor data from any of sensors 120-2 through 120-N, e.g., instead of or in addition to sensor data from sensor 120-1. The model 108 can be trained by the form training engine 110 using a variety of training techniques to improve the accuracy of inference tasks performed by the model. These training techniques can include supervised and unsupervised learning. The models can include any form of boosting techniques such as gradient boosting but can include deep learning techniques for to perform an inference task. For example, an inference task for the model 108 can include generating scores, labels, or some combination thereof indicating an evaluation of an athlete’s form. An inference task can also include generating, based on the evaluation of an athlete’s form, one or more form adjustments to improve the athlete’s form. In some examples, a model performs hybrid-learning techniques to improve accuracy of model output. Training processes for the model 108 can include any number of iterative processes, each performing a number of iterations to train the model to achieve a target performance value, e.g., an error rate below a threshold value, a generated classification label that matches the ground truth label.

[0064] The training of the model 108 can be performed using obtained ground truth data that includes known labels, associations, classifications, etc., coupled with a corresponding input, e.g., some or all of the sensor data. The form training engine 110 can adjust one or more weights or parameters of the model 108 to match estimates or predictions from the to the ground truth data, e.g., sensor data captured of an athlete performing an exercise with a sled using proper form. In some implementations, the model 108 includes one or more fully or partiallyconnected layers. Each of the layers can include one or more parameter values indicating an output of the layers. The layers of the model 108 can generate outputs for which the model can use for performing one or more inference tasks. The model 108 can be validated and tuned through holdout and test techniques, model comparison, and model selection.

[0065] FIG. 1A also shows a view 140 of the sensors 120 of the system 101 capturing sensor data of the athlete 102 performing the exercise action with a football training sled 104-1, e.g., an example of strength training device 104 that can be lifted, pushed, pulled, dragged, flipped, among other types of motions. The sled 104-1 can include optionally include additional sensors. The view 140 shows each of the sensors from sensors 120 capturing a corresponding set of sensor data, e.g., sensor 120-1 capturing sensor data 128-1, sensor 120-2 capturing sensor data 128-2, and sensor 120-N capturing sensor data 128-N. The sensors 120 can be configured to capture sensor data of the athlete 102 performing the exercise action 106 with the football training sled 104-1. The sensor data 128 for a sensor 120, e.g., sensor data 128-1 for sensor 120-1, can capture measurements and other types of characteristics of the form of athlete 102. Examples of sensors can include pressure sensors, image and / or video sensors, infrared sensors, radar sensors, force sensors, motion sensors, accelerometers, gyroscopes, etc.

[0066] For example, the athlete 102 can have a pose 109 that can have multiple phases prior to, during, and after the athlete 102 performs exercise action 106 with football training sled 104-1. As shown in view 140 FIG. 1A, the exercise action 106 can include the athlete 102 lifting the football sled 104-1, and the sensors 120 can sensor data of the pose 109 while the athlete performs the lifting exercise. In some cases, as shown by the source 122 and the sensor 120-N, the source 122 can generate and transmit a transmit signal 126-N in the environment of the athlete 102 to capture sensor data of pose 109. The sensor 120-N is configured to detect signals resulting from the transmit signal 126-N as sensor data 128-N. The sensor 120-N is to generate detections or sensor measurements based on the received signals resulting from transmit signal 126-N from source 122.

[0067] The system 101 can include and / or be in data communication with one or more sensors, e.g., a sensor from the sensors 120. For example, sensor 120-2 can be an external component in data communication with system 101. In some cases, a sensor from sensors 120 is a camera of a user device 130. A sensor can be an integrated component of system, e.g., a component of a sensing subsystem 124 such as sensor 120-N of the system. In some examples, the computing device 130-1 can receive data from a sensor 120 of an additional user device, e.g., user device 130-2 (not illustrated), data from a sensor 120-N of a sensing subsystem 124, or data from both.

[0068] A sensor from sensors 120 can include one or more hardware devices capable of receiving input from the environment and converting that input into data accessible by computing elements. For example, the one or more sensors 120 can capture image data, point cloud data, or another format of data capturing two-dimensional (2D) or three-dimensional (3D) information of the training environment, e.g., including an athlete. The sensor 120 can capture temporal data of the training environment, e.g., to capture exercise actions performed by an athlete over time.

[0069] The system 101 can include and / or be in data communication with one or more signal sources, e.g., source 122. The source 122 can generate electromagnetic radiation in one or more frequency bands. The electromagnetic radiation can be directed into the training environment, e.g., localized, and the reflections off of surfaces of the training environment, e.g., off of the athlete 102, the strength training device 104, the floor, etc., can be detectable by sensor 120-1 to capture information about a training environment surrounding the system. For example, the reflections detected by the sensor 120-1 can be used to extract pose information of an athlete within the training environment.

[0070] Source 122 can be, for example, a radio frequency (RF) source, micro wave source, acoustic signal source, laser sources, e.g., for LIDAR, or another non-visible electromagnetic radiation source. For example, a source 122 can be a millimeter (mmWave) frequency source, e.g., between about 3-30 GHz. In another example, a source 122 can be a Wi-Fi frequency source, e.g., between about 2.5-6 GHz. The source 122 can be configured to generate low power signals, e.g., 1000x lower than Wi-Fi signals, or otherwise be compatible with the training environment, e.g., low-harm to humans present in the training environment. A frequency band of the source 122 can be selected based in part on a distance of detection, e.g., distance of athlete to the system, a degree of precision of the detection, a type of sensor 120-N performing the detection, compatibility with the environment, or other design considerations.

[0071] In some implementations, source 122 can be configured to scan the training environment around the system 101. The source 122 can be configured to emit electromagnetic radiation signal in a localized direction and to scan over a range of positions in the training environment, e.g., in an arc sweeping the area surrounding the system 101. For example, the source 122 can be a laser source of a LIDAR system coupled to a scanner such that the laser is scanned across the training environment and reflected signal from the training environment is captured by the sensor 120. In another example, the source 122 is a mmWave source coupled to a scanner, e.g., a mechanical positioning apparatus, such that the mmWave frequencies arescanned about the training environment and reflected from the training environment is captured by the sensor 120. In such cases, the collected sensor data can include both temporal and spatial resolution.

[0072] In some implementations, sensor 120 and source 122 can be integrated into a subsystem, e.g., a sensing subsystem 124 where the sensing subsystem 124 can transmit frequencies into the training environment surrounding the system 101. The sensing subsystem can detect a spatial resolution of the reflections as well as a temporal resolution of the detected reflections, e.g., measure intensity, direction, etc., of the reflected signal, to generate point cloud data representing a state of the training environment.

[0073] In some implementations, sensing subsystem 124 includes one or more sensors 120 and one or more sources 122. For example, two or more sensors 120 and a source 122. In another example, two or more sensors 120 and two or more sources 122. In another example, two or more sources 122 and a sensor 120. In some instances, the sensors 120 of the sensing subsystem 124 are arranged with respect to the training environment to detect interruptions of a signal emitted by source 122 that is arranged on or in proximity to the system 101, e.g., interruptions caused by an object or person located between the source 122 and the sensors 120. In some instances, the source 122 of the sensing subsystem 124 is arranged with respect to the training environment and the one or more sensors 120 are arranged on or in proximity to the system 101 to detect interruptions in the emitted signal from the source 122, e.g., interruptions caused by an object or person located between the source 122 and the sensors 120.

[0074] In some implementations, the source 122 is a component of the user device, e.g., a cellular phone emitting a source signal. For example, the user device can emit Wi-Fi signal, Bluetooth signal, or another data communication signal. The sensors 120 of the sensing subsystem 124 can be arranged on or in proximity to the system 101 and can be configured to detect the emitted signal from the source 122. A strength, quality, or direction of the detected signal emitted by the user device can be detected by the sensors 120 and information of the position of the player and / or objects in the training environment can be inferred.

[0075] In some implementations, sensor 120 is a camera, e.g., CMOS, CCD, or other device capable of capturing image data and / or video data. The sensor 120 can be configured to capture image data and / or video data in visible wavelengths, infrared wavelengths, or other spectra.

[0076] In some implementations, sensor 120 is a pulsed coherent radar configured to transmit pulses, e.g., a transmit signal, and process returns from reflected or refracted pulses, e.g., areceive signal, from an object, e.g., the strength training device 104. The sensor 120 can be configured to maintain signal phase between the transmitted and received pulses to coherently integrate sensor measurements resulting pulses received from the object to determine object characteristics, e.g., range, velocity, acceleration, position, and other types of object state information. For example, sensor 120 can analyze phase shift between two or more successive pulses to determine a Doppler frequency shift corresponding to a relative velocity of the object, e.g., strength training device 104. The sensor 120 In some cases, the sensor 120 is configured to adjust the pulse repetition frequency (PRF), e.g., number of pulses transmitted per second, to achieve a target value for measurement accuracy (e.g., velocity) and a maximum unambiguous range for detections by the sensor 120.

[0077] The sensor 120 can be configured to apply pulse compression techniques, e.g., frequency and / or phase modulation, to increase the sensitivity of sensor 120. The increase in sensitivity for sensor 120 can provide an increase in range resolution with lower power consumption, compared to transmitting signals without pulse compression. Improvements in range resolution provide increased detectability and tracking of objects by the sensor 120. The sensor can be configured to apply signal processing techniques to reduce ambiguity in range and velocity measurements for a detected objection, thereby increasing accuracy in tracking the form of an athlete while performing an exercise with strength training device 104. In some implementations, the computing device 130-1 can be configured to control and / or adjust operation of the sensor 120, e.g., adjust PRF, modulate signals, generate waveforms. In some cases, instead of or in addition to the sensor 120, the computing device 130-1 can be configured to process received signals and apply signal processing techniques to estimate state information for an object, e.g., trajectory, rotation, position, velocity, acceleration.

[0078] In some implementations, sensor 120 is a detector capable of capturing non-visible electromagnetic radiation signal data, e.g., point cloud data. Sensor 120 can be a detector configured to capture signal in one or more frequency bands, including, for example, millimeter wave (mmWave), radio frequency (RF), Wi-Fi signal bands (2.4 GHz), frequency modulated continuous wave (FMCW) radar (77-81 GHz), or other frequency bands. For example, sensor 120 can be capable of capturing signal data reflected from (and / or transmitted through) an athlete’s body and capturing pose information of the athlete.

[0079] In some implementations, sensor 120 is an acoustic sensor capable of capturing acoustic signal data. For example, sensor 120 can be an ultrasonic sensor, e.g., a narrow-beam ultrasonic sensor, configured to capture ultrasonic frequencies reflected from an athlete’s bodyand capturing pose information of the athlete using the reflected ultrasonic frequencies. An ultrasonic source 122, e.g., an ultrasonic transducer, can be used to detect objects at distance of detection, e.g., up to about 25 meters away from the source. In some examples, an ultrasonic sensor can be integrated with a source, where the transducer of the source functions as a microphone to receive and send the ultrasonic waves and measures the distance to a target by measuring a time lapse (e.g., time of flight) between sending / receiving the ultrasonic pulse. Ultrasonic waves can have the advantage of being able to detect clear or transparent objects, e.g., liquids, and can be stable over a large array of surface finish, material, and color. Ultrasonic-based sensing subsystems can be insensitive to ambient lighting conditions. In some examples, a narrow beam ultrasonic sensor can be used to provide increased resolution within a distance range useful for the training environment. In another example, sensor 120 can be configured to capture low-level acoustic signals reflected from and / or transmitted through an athlete’s body and capturing pose information of the athlete using the low-level acoustic signals.

[0080] In some implementations, the system 101 includes distance sensors such as a light detection and ranging (LIDAR) sensor, a time-of-flight sensor, etc. The data, images, and / or video captured by the image sensor may also be processed to reduce the noise and / or vibration in the signal using. In some implementations, sensor 120 collects pre-data which the system 101 can process to extract biometric data and / or pose information for the athlete. For example, the sensor 120 collects point cloud data from which the system 101 may pre-process, e.g., filter, and extract biometric data for the athlete. In some implementations, the system can pre-process the sensor data to reduce signal-to-noise, filter, or otherwise clean up the sensor data prior to extracting the action data. The system can pre-process the data by flattening 3D sensor data to a 2D space. In some implementations, sensor 120 is a radar sensor, e.g., mmWave sensor, collecting radar data.

[0081] Sensor 120 can be integrated with or in data communication with additional processing components, e.g., hardware accelerators, digital signal processors, or the like, which process the radar data to generate further processed data forms. For example, radar data can be analog-to-digital (ADC) data output from the sensor 120, e.g., frames where each frame is a representation of collected chirps for the period of time of the frame (framerate). The radar data can be further processed to produce refined data, e.g., point cloud heat maps, using low-level radar processing. A further refinement of the data can include a first high-level radar processing to produce target tracking and motion / object decisions, e.g., using target1information extraction. Another further refinement of the data can include a second high-level radar processing to produce target classification, e.g., using classification information and / or micro-Doppler effects. The additional processing components can generate, from the refined radar data, a visualization including scene interpretation from which form training process described herein can proceed.

[0082] In some implementations, the system 101 includes two or more sensors 120 of a same or different type. In some examples, the system 101 includes at least one camera and at least one non-visible electromagnetic radiation sensor. In such examples, the sensors 120 can collect multi-modal data input, e.g., data originating from two or more different types of sensors. For example, image data and point cloud data, which can be used collectively by the system to generate form training information.

[0083] The sensor 120 can be mounted such that the field of view of the sensor 120 is capable of capturing images of an athlete when the athlete approaches the football sled 104-1 to prepare for an exercise action 106. In some cases, this can involve a co-located sensor 120-1 or a remote sensor 120-2, e.g., relative to the computing device 130-1, and / or a sensor 120-3 mounted on the football sled 104-1. As an example, an image sensor is can be an integrated sensor 120 of a user device 130-1, e.g., cellular phone with a camera, in data communication with the system 101. In some examples, a sensor 120, e.g., sensor 120-N, is a component of a sensing subsystem 124 of the system 101.

[0084] In some implementations, a sensor 120 (e.g., one or sensors from sensors 120-1 through 120-N) can be an image sensor configured to image data for facial recognition, e.g., by the image sensor and / or the user device 130-1. The image sensor can be a camera, a depth sensor, e.g., an infrared sensor, a time-of-flight sensor to measure light reflection, or some combination thereof, to capture sensor measurements for facial recognition of a user of the strength training form training system 101. The sensor 120 and / or the user device 130-1 can be configured to perform facial recognition of a user of the strength training device 104 to identify the user to a profile for their workouts, e.g., performance of exercise actions. The system 101 can automatically log, e.g., store, sensor measurements, action profiles, and other types of collected data for the identified user.

[0085] In some implementations, a sensor 120 can be a Global Positioning System (GPS)-enabled sensor to capture sensor measurements to determine position, speed, distance, and other types of state information. For example, a GPS-enabled sensor 120 can provide real-timeor near real-time measurement of state information for the strength training device 104 and / or the athlete 102.

[0086] FIG. IB is a diagram of the example exercise actions using a sled of the system. FIG. IB shows a view 150 with the athlete 102 performing two different exercise actions 106-1 and 106-2. For example, view 150 shows the athlete 102 performing exercise action 106-1, which includes the athlete 102 approaching the strength training device 104 and then lifting the strength training device 104, e.g., to flip the strength training device 104. The system 101 can obtain sensor data capturing pose information of the athlete 102 as the athlete 102 approaches the sled, such as the stance / posture of the athlete 102, the foot placement of the athlete 102, the grip type and placement applied by athlete 102 to make contact with the strength training device 104, etc. Similarly, the system 101 can obtain sensor data, e.g., by capturing sensor measurements using sensors 120, of the athlete 102 as they perform the exercise action, which can include the pose of the athlete 102 while lifting and / or pushing the strength training device 104. As another example, the system 101 can be configured to capture sensor information representing pose of the athlete 102 and other physical characteristics after performing the exercise action 106-1. In this way, the system 101 can be configured to determine a form of the exercise action 106-1 performed by the athlete 102 and determine one or more form adjustments to improve the form of the athlete 102.

[0087] As another example, the system 101 can be configured to capture sensor data of exercise action 106-2. The view 150 shows athlete 102-1 and athlete 102-2 (collectively “athletes 102”) performing an exercise action 106-2 with the strength training device 104 together. The exercise action 106-2 can be an exercise to push or pull the strength training device 104 across a surface of the training environment, e.g., referring to training environment 100. The system 101 can be configured to capture multiple sets of sensor data for each of the athletes 102 performing the exercise action 106-2. For example, the system 101 can be configured to determine a form for each of athlete 102-1 and 102-2 based on their respective forms prior to, during, and / or after performing the exercise action 106-2.

[0088] The strength training device form training system, e.g., system 101, can be configured to capture sensor measurements of the athlete’s poses, stances, grip, and other aspects of form to perform the exercise action 106. Exercise actions can include dragging and pulling the sled. For example, the athlete can drag and pull the sled, e.g., the athlete lifts the sled at an angle relative to a ground using an underhand grip, and then the athlete pulls the sled along the ground surface at the angle while moving backward. In another example, the athlete can perform aclean and press (e.g., a full-body exercise that engages multiple muscle groups like the legs, core, and upper body of the athlete) using the sled. The athlete performs the exercise action by first performing a clean from an initial posture and stance, e.g., lifting one end of the sled using an overhand grip to lift the sled from the ground at an angle, performing a press with the sled, e.g., pushing the sled away in a horizontal direction away from the athlete, and then returning to the initial posture and stance. In some cases, rather than performing a clean to lift the sled to a shoulder height, the athlete can perform a standing press to lift the sled and can perform repetitions of a press, e.g., lifting the weight overhead from shoulder level, to perform an exercise that focuses on training upper body muscle groups.

[0089] An example of an exercise action 106 can also include a sled push, where the athlete adopts a low, forward leaning posture and stance (e.g., bending the knees slightly and angling the athlete’s chest towards the sled) and uses an overhand grip to hold a top portion of a sled, e.g., a handle. The athlete can then perform an initial drive, by pushing their feet into the ground and then extending their legs to generate an amount of force that moves the sled. The athlete can then continue applying force to keep the sled in momentum until the athlete releases the sled and reverts to standing posture and stance. The athlete can then repeat the exercise, e.g., according to a workout training program.

[0090] The athlete can also perform exercise actions that are tailored towards particular sports or applications. In the context of American football, the athlete can perform a lineman throw using the strength training device by approaching the strength training device from a kneeling position. The athlete can then adopt an overhand grip at a first end of the strength training device, lift the strength training device to shoulder level while maintaining the kneeling position, switch to an underhand grip, and push the strength training device such that the first end rotates around a point located at a second end of the strength training device, the second end being opposite from the first end. In this way, the athlete can mimic explosive pushing and throwing actions in a game of American football by performing exercise actions, with the system 101 continually monitoring the athlete’s form and providing feedback to reduce the risk of injury, increase exercise effectiveness, among the other benefits of form improvement training.

[0091] The athlete can use a strength training device (e.g., such as a sled with or without additional weights) to perform exercise across different applications such as general or recreational fitness, e.g., to improve one’s day-to-day strength and endurance or for recreational activities / sports, as well as applications in first responder physical training, e.g.,for lifting debris, carrying people and / or equipment for rescue, firefighting, etc. The athlete can perform exercises such as a tire flip, barrel rolls, deadlifts, among others. For example, the athlete can perform a tire lift by first adopting a stance where the athlete’s feet are should-width apart, performing a squat by bending their knees, and gripping the strength training device to perform a lift, e.g., by drive their feet into the ground to push one end of the strength training device upward. After lifting the strength training device, e.g., to chest or shoulder height and standing from the squat position, e.g., by extending knees with a straight back, the athlete can push the strength training device forward to flip the strength training device.

[0092] The athlete can perform a barrel roll exercise using the strength training device. For example, the athlete can grip a left portion or a right portion of a first end of the strength training device, e.g., using an underhand grip, with one or both hands. After gripping the strength training device, the athlete can pull the strength training device along a direction perpendicular to a center axis of the strength training device. In this way, the athlete flips the strength training device in along the perpendicular direction. The athlete can repeat the exercise using one or both hands to flip the strength training device in the same perpendicular direction again or can flip the strength training device in a direction opposite to the perpendicular direction, e.g., to return the strength training device to its original position. This is a particularly useful exercise in circumstances when space is limited and angles of approach to the object being lifted are limited, such as a piece of debris that can only be approached from a narrow range of angles, e.g., by a first responder.

[0093] As another example, the athlete can perform a deadlift using the strength training device (e.g., a sled), which includes the athlete using an overhand grip on a handle on one end of the strength training device while the athlete is in a squat position, then lifting the strength training device upward from the ground using a pulling motion at the end being held. The athlete can shift from a squat position to a standing position when lifting the strength training device and then lower the strength training device back to the ground while returning from the standing position to the squat position. Although this specification describes different exercises using the strength training device, the strength training device form training system 101 can applied to combinations of exercises. For example, the system 101 can identify and improve the form of an athlete performing any workout using any combination of exercises, e.g., a workout that mixes, repeats, re-orders different exercises using the strength training device.

[0094] FIG. 2 is a block diagram 200 of system 101 communicatively coupled with server 201 that executes workout module 202 to generate a workout program executed by system 101. Asillustrated in FIG. 2, server 201 includes workout module 202 and application environment 203. Workout module 202 includes database 204, which includes accounts 206, workouts 208, and analytics 210.

[0095] Server 201, as illustrated in FIG. 2, can be a cloud-based or otherwise remote server computer including one or more processors and computer-readable memory encoded with instructions that, when executed by the one or more processors, cause server 201 to execute workout module 202 and application environment 203 according to techniques described herein. Examples of server 201 include, but are not limited to, mainframe computers, desktop computers, laptop computers, or other computing devices capable of implementing workout module 202 and managing operation of (e.g., serving of) application environment 203. In some examples, rather than a single server device, server 201 may be implemented as a server system including multiple interconnected server computers that distribute functionality attributed herein to server 201 among the multiple server computers. For instance, in certain examples, server 201 can be implemented as a server system including a first server computer that executes workout module 202 and a second server computer that executes (e.g., serves) application environment 203. Similarly, while database 204 is illustrated as included in workout module 202 of server 201, database 204 can include any one or more databases that may be local to server 201 or distributed among any one or more server devices operatively connected to server 201.

[0096] Workouts 208 store workout programs that include machine workout instructions that are executed by system 101 to provide workout / exercise action instructions to the athlete to improve the athlete’s form for performing the exercise. Workouts 208 are associated with attributes, such as a workout skill level, workout intensity level, workout time, workout type (e.g., offensive skills development, lifting form, grip type and placement, foot placement, agility development, strength development, physical conditioning development, or other workout type), or other attributes.

[0097] Workout programs stored at workouts 208 can be associated with any one or more accounts and / or account groups stored in database 204 at accounts 206. For instance, a particular workout program can be associated with (e.g., assigned to) an account group corresponding to a team, and therefore also associated with each individual user account that is a member of the team account group through the hierarchical relationship between the parent team account group and the child user accounts. Workout programs stored at workouts 208 can be associated with a single account of accounts 206 or multiple accounts of accounts 206.As such, workout programs can be generated and stored at workouts 208 of database 204 and utilized by a single user account or shared between multiple user accounts or account groups.

[0098] Analytics 210 of database 204 store analytics data (e.g., statistics) associated with any one or more accounts stored at accounts 206 and / or workout programs stored at workouts 208. Examples of analytics data include form training data (e.g., exercise profiles), amount of weight lifted, an amount of time to perform the exercise action, a usage time of system 101, user heart rate data during any one or more workout programs (e.g., sensed by a heart rate monitor or other physical monitoring device worn by an athlete during a workout program), shooting percentage relative to heart rate, movement, position relative to the sled, or other analytics data. Analytics data stored at analytics 210 of database 204 can be associated with a workout program, such that each user account that executes a particular workout program contributes to shared analytics corresponding to the executed workout program.

[0099] Analytics 210 can store any statistical or other analytical data that corresponds to user accounts, user account groups (e.g., team account groups), and workout programs to enable comparison of performance between user accounts, between user accounts and benchmark performance criteria, between time-separated performances of a single user account (or account group), or other comparisons. As such, analytics data stored at analytics 210 can enable a coach, athlete, or other user to track performance of a single athlete or group of athletes over time, to compare performances between athletes or groups of athletes, and to track progress of skill development and conditioning of athletes or groups of athletes. For example, analytics 210 can enable a coach, athlete, or other user to track user form compared to an ideal form and / or a self-consistent form for multiple different exercise actions, e.g., lifting, pushing, pulling, flipping, over time and between a same session or two or more different sessions.

[0100] As illustrated in FIG. 2, server 201 can execute an application environment 203, for example, through a web browser accessible on a user device 130 or in a local application located on the user device 130. User devices 130 can be, for example, tablet computers, mobile phones (including smartphones), laptop computers, wearable devices such as smartwatches, or other computing devices that can communicate with server 201 to interface with workout module 202. Application environment 203 is communicatively coupled with workout module 202 to provide a user interface that enables user interaction with workout module 202 to create and select workout programs and view analytics data stored at database 204, as is further described below.

[0101] system 101 is communicatively coupled with server 201 to access the application environment 203 via any one or more wired or wireless communication networks, such as a cellular communication network, local area network (LAN), wide area network (WAN) such as the Internet, wireless LAN (WLAN), or other type of communication network. In operation, an athlete utilizes user device 130 (e.g., a smartphone) to execute the application environment 203 through an application (e.g., an app) that interfaces with workout module 202 or to access the application environment 203 via a web browser that presents a graphical user interface managed by workout module 202. The graphical user interface presents a login screen that enables the athlete to provide account login information, such as username and passcode. Workout module 202 accesses the account stored in accounts 206 associated with the login information (or enables the athlete to create a new account) and presents the user with graphical control elements to either select a workout program stored at workouts 208 of database 204 or create a new workout program, as is further described below.

[0102] The system 101, as illustrated in FIG. 2, includes an interface (illustrated as I / F), such as a console, a touchscreen interface, a keyboard and / or mouse interface, or other type of interface to enable user interaction with application environment 203 to create a workout program and / or select a workout program stored at workouts 208 for execution by system 101.

[0103] In operation, a user selects, through the graphical user interface of application environment 203, a workout program stored at workouts 208 and / or create a new workout program via the interface provided by application environment 203 and managed by workout module 202. Server 201 transmits the selected or created workout program to system 101. The workout program includes athlete workout instructions representing athlete activity during workout program. The system 101 can help the athlete execute the athlete work instructions by providing form adjustments for the athlete performing the exercise.

[0104] The system 101 presents, through a graphical user interface of the application environment 203, athlete instructions for review prior to execution of the workout program and, in certain examples, presents the athlete instructions during execution of the workout program via a display, speakers, or other output device. Results of the workout program corresponding to proper and improper form while performing exercising actions, a scoring indicating a deviation of the form, adjustments to the exercise form, duration of one or more portions of the workout program, or other analytics data can be transmitted by system 101 to workout module 202 of server 201. Accordingly, the system 101 enables a user to select one or more workout programs stored at workouts 208 of database 204, create a new workoutprogram that can optionally be stored in workouts 208, and execute the workout program to enable effective training for the athlete.

[0105] A user of the system 101, e.g., an athlete, coach, or other interested part, can interact with the system 101 through a graphical user interface (GUI) of the application environment 203, that receives user input and provides feedback to the user through the graphical user interface. As described in further details below, the graphical user interface can be presented in an application environment on a user device 130, e.g., an athlete’s mobile device, or in display of a console of the system 101. A user can interact with the system 101 through the graphical user interface, for example, to define a workout program that includes the particular exercise actions to be performed by a user and to receive feedback related to a workout program in progress and / or completed, e.g., form training.

[0106] The system 101 presents, through the graphical user interface, graphical control elements that enable the user to interact with the system 101, e.g., to select a type of exercise action to perform. The graphical user interface can be managed by a server device, communicatively coupled with the system 101 or a separate computing device, e.g., a user device 130, to receive the workout program including the athlete workout instructions presented to the user.

[0107] As such, the system 101, through to the graphical user interface, enables a user (e.g., an athlete, coach, administrator, training expert, or other user) to define a workout program via the graphical user interface of the system 101 for execution and, in certain examples, enables selection of desired different exercise actions that are not limited by indications of an original exercise action. For example, a form of one type of exercise action can be used to provide form adjustments of a different type of exercise action that can share one or more motions, e.g., sharing a same type of grip, foot placement, and other exercise factors. In this way, the system 101 allows the generation of workout programs by differing users via one or more computing devices that are communicatively coupled with a server device that manages operation of the graphical user interface.

[0108] Though described with reference to FIG. 2 as a system 101 in data communication with a system 101, in some implementations, the system 101 provides the form training processes described in this specification without using some or all of the functionality of the system 101. In some examples, the system 101 can collect captured sensor data using a standalone sensing subsystem 124 and / or from a sensor 120 of a user device 130, where the captured sensor data includes the athlete performing the exercise action. In such cases, thesystem can provide guidance to the user to collect the sensor data of the athlete performing the exercise action using the strength training device 104. The guidance can include positioning of the camera of the user device 130 or components of the sensing subsystem 124 with respect to an athlete’s position and the strength training device 104. The system can receive the collected data from the user device 130 and / or the sensing subsystem 124 and process the collected sensor data at server 201 to proceed with the form training process, as described herein.

[0109] The system 101 is configured to provide control commands to components of the system 101. For example, system 101 is configured to provide control commands to sensing subsystem 124.

[0110] In some implementations, system 101 can receive, e.g., from a user on a user device 130, a workout program selection. The workout program includes indications of different instances and / or types of exercise actions, such as drills for football training and positional information display via a visual representation presented by a graphical user interface executed by, e.g., a remote server device. The system 101 provides control signals to components of the system 101 to capture sensor data, e.g., using sensors 120, of the athlete 102 performing the exercise action 106.

[0111] Referring to computer-readable memory of the user devices 130, the sensing subsystem 124 and other connected devices can be configured to store information within devices during operation. Computer-readable memory, in some examples, is described as computer-readable storage media. In some examples, a computer-readable storage medium can include a non-transitory medium. The term “non-transitory” can indicate that the storage medium is not embodied in a carrier wave or a propagated signal. In certain examples, a non-transitory storage medium can store data that can, over time, change (e.g., in RAM or cache). In some examples, the computer-readable memory is a temporary memory, meaning that a primary purpose of the computer-readable memory is not long-term storage. Computer-readable memory, in some examples, includes volatile memory that does not maintain stored contents when electrical power to a computing device, e.g., a user device 130-1, is removed. Examples of volatile memories can include random access memories (RAM), dynamic random access memories (DRAM), static random access memories (SRAM), and other forms of volatile memories. In some examples, computer-readable memory is used to store program instructions for execution by the one or more processors, e.g., of user device 130-1, sensing subsystem 124. For instance, computer-readable memory, in some examples, is used bysoftware or applications running on a computing device to temporarily store information during program execution.

[0112] Computer-readable memory, in some examples, also includes one or more computer-readable storage media that can be configured to store larger amounts of information than volatile memory. In some examples, computer-readable memory includes non-volatile storage elements. Examples of such non-volatile storage elements can include magnetic hard discs, optical discs, floppy discs, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories.Form Training Process

[0113] FIG. 3A is a flow diagram of an example process 300 of the sensor-based system. Briefly, the process 300 includes obtaining, from the at least one sensor, sensor data including a person performing an exercise action with the sled (302), determining, from the sensor data, an exercise form including one or more points tracking the exercise action performed by the person, each point from the one or more points representing a location on the person (304), and generating, by a model and for the exercise form, an evaluation indicating a deviation of the exercise form from an ideal form for performing the exercise action (306).

[0114] The system obtains, from the at least one sensor, sensor data including a person performing an exercise action with the sled (302). For example, and as described in reference to FIG. 1A above, a sensor 120-1 from sensors 120 can be configured by the system 101 to capture sensor data 128-1 of the athlete 102 performing the exercise action 106 with sled 104-1. In some cases, the sensor data can capture information related to pose 109 of the athlete 102 prior to, during, and / or after performing the exercise action 106. The multiple sensors 120 can be the same or different types of sensors, e.g., a camera and a mmWave sensor, each capturing respective sensor data.

[0115] The system determines, from the sensor data, an exercise form including one or more points tracking the exercise action performed by the person, each point from the one or more points representing a location on the person (304). As described in reference to FIG. 7 below, the process can include determining a form based on one or more biomechanical points on the athlete’s body.

[0116] The system generates, by a model and for the exercise form, an evaluation indicating a deviation of the exercise form from an ideal form for performing the exercise action (306).As described in reference to FIG. 1A above, the system 101 can be configured to generate, by model 108, an evaluation indicating a deviation from good form for performing the exercise action 106.

[0117] In some implementations, the system obtains the sensor data from two or more different types of sensors, e.g., multi-modal sensor data, from the same or different angles of the same shot action (or set of shot actions). For example, the system can obtain video data and point cloud data from respective sensors of the shot action (or set of shot actions) captured by respective sensors.

[0118] FIG. 3B is a flow diagram of an example process 350 for collecting and indexing pose information for an athlete using a system 101. The system initiates a workout program (352). The workout program can include a series of different exercise actions, a number of sets and reps for an exercise action, other types of drills for training, or some combination thereof. The system can initiate the program in response to receiving a selected workout program from a user, e.g., the athlete or their trainer. The system can provide an audio and / or visual representation of the workout program to the user, e.g., through the graphical user interface of the application environment 203, to provide guidance before, during, and after the workout program is run.

[0119] The system obtains, from sensor, sensor data capturing a person performing an exercise action (354). For example, the sensors 120 of the system 101 can capture sensor data of the athlete. As described above, the sensor can be, for example, a camera of the user’s device 130 having a field of view of the user as the user is performing the workout program. In some examples, the sensor can be configured to capture sensor data in a frequency band of the electromagnetic spectrum outside the visible frequency band. The sensor 120 can be an integrated component of the system 101 or a component of a user device 130 in data communication with the system 101. At times, multiple sensors 120 can be used to capture sensor data of the exercise action simultaneously or sequentially. A position of the field of view of the sensor can be specific to a target angle, e.g., face-on, side-view, or another angle, of the user’s form with respect to the system 101.

[0120] In some implementations, the system obtains the sensor data, e.g., video data including the multiple frames, by capturing sensor data including the person performing the exercise action two or more times. The system can capture sensor data simultaneously from two or more different angles of a same exercise action (or set of exercise actions), e.g., using the same ordifferent sensors 120. Alternatively, or additionally, the system can capture video data sequentially of the user performing the exercise action from two or more different angles.

[0121] In some implementations, the system 101 assists, through the graphical user interface, a user in positioning the one or more image sensors, e.g., a camera of a user device 130, with respect to the system 101 and / or the strength training device 104 to capture video data of the user performing a selected workout program, e.g., for form training. For example, as depicted in FIG. 8, the graphical user interface 800 can present a visual indicator 802, e.g., a highlighted region, outlined region, arrows, or other indicators, within a display 804 of a real-time capture of image data from the camera 120 of the user device 130 to guide the user on where to position the user device 130 in order to capture one or more angles of the user performing the workout program. For example, the graphical user interface can guide the user to position the camera of the user device to capture a face-on view, a side-view, or an angled view of the user when the user is positioned with respect to the system 101 during a workout program.

[0122] In some implementations, the system 101 assists, through the graphical user interface, a user in positioning themselves with respect to one or more sensors, e.g., sensors of the sensing subsystem 124 of the system 101. For example, the graphical user interface can present visual cues for the user to follow to align themselves within the field of view of the integrated sensor of the system.

[0123] In some implementations, obtaining the sensor data, e.g., video, includes instructing, by the system and through the graphical user interface, e.g., as depicted in FIG. 8, the user to move a location of the image sensor (e.g., the camera of the user device) from a first video data capture location to a second video data capture location, e.g., to capture video data of the user performing the exercise action from multiple angles.

[0124] In some implementations, obtaining the sensor data, e.g., point cloud data, includes instructing, by the system and through visual cues to the user, the user to move to a different location, e.g., from a first sensor data capture location to a second sensor data capture location, with respect to the sensor to capture sensor data of the user performing the exercise action from multiple angles.

[0125] FIG. 6 is a flow diagram of an example process 600 for a strength training device form training system. For convenience, the process 600 will be described as being performed by a system of one or more computers, located in one or more locations, and programmed appropriately in accordance with this specification. For example, the strength training device form training system 101 of FIG. 1 A, appropriately programmed, can perform the process 600.Additionally, although the example process 600 is described with respect to an image sensor and captured video data, the processes described with reference to FIG. 6 can be performed by one or more sensors configured to capture respective sensor data, where at least one of the one or more sensors is configured to capture sensor data in a non-visible wavelength spectrum, e.g., mmWave, Wi-Fi, or the like.

[0126] The system provides (602), for presentation in a user interface, a first visual indication of a first position to arrange the image sensor with respect to the person performing the exercise action to capture the video data. For example, the system can guide the user to position the user device in a first location to capture a first angle of the user’s form performing the exercise action(s), e.g., face-on, for a first set of exercise actions, and subsequently the system can guide the user to re-position the user device in a second location to capture a second angle of the user’s form performing the exercise action(s), e.g., side-view for a second set of exercise actions. Some or all of the guidance can be represented visually in the graphical user interface to assist the user in aligning the field of view of the image sensor to capture the correct angle of the user’s form. For example, the graphical user interface can include a target box, arrows, or other indicators, highlighting a location of where a user’s body should be located during the capture of the video data, e.g., as depicted in FIG. 8.

[0127] The system receives (604), from the image sensor, the video data, and determines (606), from the video data, that a threshold video data capturing the exercise action from the first position is met. For example, the system can determine that a threshold number of exercise actions are captured from the first position. A threshold for the number of exercise actions can depend, in part, on an amount of biometric data points extractable from the location, e.g., if some of the biometric data points are obscured or not discernable by the pose information model.

[0128] The system provides (608), for presentation in the user interface, a second visual indication of a completion of capture of the video data from the first position. The system can provide visual, audio, and / or haptic feedback to the user to indicate that the portion of the workout program is completed and to move on to the next step or conclude the workout program.

[0129] The system provides (610), for presentation in the user interface, a third visual indication of a second position to arrange the image sensor with respect to the person performing the exercise action to capture the video data, where the second position is at leastone of (i) a different position, or (ii) at a different angle, with respect to the strength training device form training system from the first position.

[0130] The system receives (612), from the image sensor, the video data and determines (614), from the video data, that a threshold video data capturing the exercise action from the second position is met. For example, the system can determine that a threshold number of exercise actions are captured from the second position. In some instances, a threshold number of exercise actions can be different between the first position and the second position. For example, the system may require fewer instances of an exercise being performed from a face-on view the exercise performed from a side-view of the user. A threshold for the number of exercise actions can depend, in part, on an amount of biometric data points extractable from the location, e.g., if some of the biometric data points are obscured or not discernable by a model for pose estimation.

[0131] The system provides (616), for presentation in the user interface, a fourth visual indication of a completion of capture of the video data from the second position. The system can provide visual, audio, and / or haptic feedback to the user to indicate that the portion of the workout program is completed and to move on to the next step or conclude the workout program.

[0132] In some implementations, the system can provide, to a trained machine learning model 108, video data capturing the exercise action from the first position, and obtain as output, from the trained machine learning model, extrapolated synthetic video data representing the exercise action from a second, virtual position.

[0133] In some implementations, the system receives sensor data from two or more sensors, e.g., multi-modal sensor data, and determines, from the sensor data that a threshold sensor data capturing the shot action from the first position is met. For example, the multi-modal sensor data can include video data captured by an image sensor and point cloud data captured by a mmWave detector.

[0134] Referring back to FIG. 3B, the system extracts, from the sensor data capturing the person performing the exercise action, action data (356). For example, the system extracts the action data from frames of captured video data including the person performing the exercise action. In another example, the system extracts the action data from multi-modal sensor data. The action data includes biomechanical pose information including multiple points, e.g., coordinates in two-dimensional or three-dimensional space, tracking the biomechanics of the exercise action. The multiple points track respective locations of biometric data pointscorresponding to the person appearing in the sensor data and tracking the exercise action. For example, as depicted in FIG. 7, a computer vision pose detection model is used to translate sensor data capturing a user’s body into a biomechanical form 700 of points, e.g., elbow points 13, 14, detecting and tracking the user’s limb positions and movements thereof before, during, and after the exercise action. The pose detection model can further be used to define relative locations between points, e.g., distance 706 between knee points 25 and 26. The pose detection model can be used to define directions, e.g., vectors, of movement, for example, vectors 708, 710 indicating directions of movement of the feet of the form.

[0135] In some implementations, extracting biomechanical pose information includes resolving the most likely arrangement of a pose rig including a data structure to represent joints and rigid elements of the human body. A solver can find, for example, the angle values for each joint most likely (e.g., with the lowest error value) to produce an outline of a human matching an outline found of the athlete. It will be appreciated that other processes for determining pose information can be used, including the use of machine-learning trained classifiers. Action data includes timing information for the exercise action with the pose information. For example, at each frame, a data object to represent the pose can be stored in a datastore and indexed based on a time value. The time value may be a 24-hour clock representing time of day, may be a time value measuring time elapsed since the start of the workout, or may be a frame number.

[0136] In some implementations, the system can access action data generated by other biomechanical tracking sensors, e.g., wearable technology, that tracks body movements, to enrich the action data extracted from the video data. The system can use the additional action data from biomechanical tracking sensors to detect, log, and reinforce positions of body points of the user over time.

[0137] In some implementations, the system integrates action data including object detection data for one or more obj ects. In other words, the action data further includes tracking respective locations of an object appearing in the sensor data, e.g., the strength training device. Tracking the respective locations of the object can include tracking the object relative to the biometric data points corresponding to the person in the sensor data and / or relative to another (e.g., stationary) frame of references. In some examples, the system can track the position of the strength training device 104 relative to the biometric points corresponding to the dynamic position of the user’s hands and / or track the position of the strength training device with a static frame of reference, e.g., the camera’s world coordinates. The object position can be tracked through space and time to extract one or more object metrics. For example, an object metricincludes an angle the object as it travels in an arc and approaches a target location (e.g., the strength training device moving from one location to another location, the position being flipped and / or lifted from one end). In another example, an object metric includes a spin rate of the object, e.g., in revolutions per unit time. In another example, an object metric includes a contact / no contact with boundaries of the location, e.g., a prediction position for the strength training device 104 after performing the exercise action with proper form.

[0138] In some implementations, the system enriches the action data with exercise completion feedback, e.g., total weight of the strength training device, success rate, time to complex exercise, power output from exercise. In some cases, the strength training device form training system may be equipped with a sensor captures measurements related to motions applied to the strength training device. These sensors can include, but are not limited to, any of the sensors described in this specification, contacting sensors such as vibration sensors, or pressure switches attached to or positioned within the strength training device. These sensors can include, but are not limited to, non-contacting sensors such as vision sensors or time of flight sensors. In still other examples, the sensors used to collect data other than video such as LIDAR data.

[0139] In some implementations, the system integrates exercise completion feedback with the action data. For example, using the techniques described in this document, exercise completion feedback for each exercise action in a workout can be created and indexed. With this information, the strength training device form training system 101 can draw correlations between the action profile of the athlete’s form and exercise outcome. For example, the strength training device form training system may determine that a particular type of form breakdown (e.g., insufficient grip) is correlated with exercises that are not successfully completed. In another example for another athlete, the system may determine that similar form breakdown is not correlated with any such inaccuracies in grip placement. For example, workout information involving the strength training device can be logged by the system and used to enrich the action data collected from the sensor data, e.g., exercise completion feedback.

[0140] The system identifies, from the action data, one or more phases of the exercise action (358). The exercise action is deconstructed into multiple phases, e.g., one or more phases, each phase of the exercise action represented by a subset of the biometric data points that track a timing and movement of the respective phase of the exercise action. Each phase of the two or more phases can be defined by coordinate points and vectors for the points through a durationof the phase of the exercise action. In some implementations, an exercise action can include one phase for the exercise action. For example, a phase for an exercise action can present a stance, pose, etc. of the user.

[0141] In some implementations, the phases can be defined by manual labeling, automated labeling, or a combination thereof, where identification of coordinate points and vectors for the points (defining the movement of the points through space) for the phase through the duration of each phase can be performed in an automatic, semi-automatic, or manual process. For example, human experts can label a sequence of exercise actions to identify the durations of each of the phases and identify the relevant points used to define the phase. In another example, one or more trained machine learning models can receive a sequence of exercise actions as input and provide, as output, predictions of coordinate points and vectors for the points corresponding to each phase of the exercise action.

[0142] In some implementations, the phases of the action profile include lifting, pushing, flipping, follow through, and landing. Each of the phases correspond to a particular movement within the exercise action where the system can use a subset of the action data extracted from the captured sensor data of the exercise action to define the start, duration, and end of the phase. Phases can be defined using respective subsets of points of the action data, distances / relative locations between selected subset of points of the action data, coordinated movements of particular subsets of points of the action data, angular relationship between limbs as defined by subsets of points of the action data, or any combination thereof. Phases can be defined by timing of absolute and / or relative movements of subsets of points of the action data. Each phase can be defined by a different measure of a respective subset of the action data.

[0143] In another example, the position phase includes a position of the athlete before the strength training device is approached for a lift and can be defined based in part on, for example, the position of the hip positions extracted from the biometric data.

[0144] In another example, a lift phase is defined at a start of the exercise action, e.g., at first contact between the athlete and the strength training device, and ends at a release point of the strength training device, e.g., as defined by an angular relationship between a subset of points. In some implementations, the lift phase can include an evaluation of grip between the athlete and the strength training device. The lift phase can also include body alignment of the athlete relative to the strength training device.

[0145] In another example, a flip path phase is defined at the start of the y-coordinate and an end point when the elbows of an athlete are substantially extended, or when the elbow passes the shoulder points.

[0146] In some implementations, at least one phase of the two or more phases overlaps in timing with at least one other phase of the two or more phases during the exercise action. FIG.9 depicts a block diagram of an example of phases of the action profile over the timing of the exercise action, e.g., phases 902, 904, 906. As depicted, a lift motion phase can overlap with at least a portion of position phase, a flip phase, and follow through phases.

[0147] In some implementations, different exercise actions can be deconstructed into different sets of phases. For example, an exercise action including a “lifting and flipping” movement can have a distinct set of phases in a corresponding action profile than an exercise action that includes a “push then pull” movement.

[0148] Referring back to FIG. 3B, the system generates, from the extracted action data, an action profile for the exercise action including the multiple phases (360). Each phase relates to an exercise action-specific movement within the exercise action that relates the pose data to a specific phase metrics. In some implementations, the system further evaluates phase metrics for each of the phases of the action profile to determine a deviation of the user’s form for the phase against a target form. A target form can be, for example, an ideal form (as defined by the strength training device form training system, experts in the field, a trainer or coach, or another authority on form) and / or a self-consi stent form defining an aggregated historical form for the particular user over a previous amount of time.

[0149] Phase metrics can be defined to mathematically define the relationships between the biomechanics of the exercise action represented by the phases of the action profile and the outcome of the exercise action (e.g., successful, unsuccessful). The phase metrics can be used to perform comparisons between the user’s action profile and an ideal action profile, and can be used to measure consistency and / or fidelity of the exercise action and make performance recommendations. Each phase can have associated phase metrics to define the aspects of the phase. At times, a phase metric can span one or more phases.

[0150] For example, a phase metric can define the start and finish of an athlete’s exercise action path and measure an efficiency of movement within the path. Equation 1 below shows the duration of an exercise action where tstartis the start time and tendis the end time of the exercise action, and v(t) is the velocity over time. An efficiency of movement can be a ratio of an amount of work done relative to an amount of energy expanded.endDuration = tend— tstart; Path Length = J ||v(t)|| dt (1) start

[0151] In another example, a phase metric can measure an athlete’s ability to keep the strength training device within a defined area of the exercise action and generates a corresponding score indicating a fidelity to the area. Equation 2 below shows the percentage of time P based on a total duration T, an object position over time xobject(t), and an indicator functionareathat equal 1 when the object is in the defined area and 0 when the object is not in the defined area:

[0152] In another example, a phase metric can measure a time of the athlete in a particular phase of an exercise action. In another example, a phase metric can measure a degree to which the athlete’s arms or legs are extended and posture during the exercise action. Equation 3 below shows an extension score E representing a degree of limb extension and position, based on a joint angle and a maximum extension angle

[0153] In another example, a phase metric can measure a fluidity and coordination of the athlete's body swaying during the exercise action, e.g., the smoother the motion, the higher the score. Equation 4 below shows a fluidity score F based on a sway displacement x(t) and a total duration T

[0154] In another example, a phase metric can measure a release time when the athlete releases the strength training device during the exercise action, e.g., a flipping action. Equation 5 below shows a release time treieasebased on a grip force Fgrip(t) and threshold for release e

[0155] In another example, a phase metric can measure a deviation between the distance that the athlete’s heels are apart when their feet leave the ground, and the distance between their heels when they touch back down on the ground. Equation 6 below shows a heel distance deviation D at takeoff htakeoffand landing hlanding

[0156] In another example, a phase metric can measure an average release time for a set of exercise actions. In another example, a phase metric can determine fastest and slowest release times of a set of exercise actions. Equation 7 below shows a fastest release time tfastest and slowest release time tsiowest, where trleiease is the release time in trial i:

[0157] The average release time can be the sum of release times for a number of trials n, divided by the number of trials.

[0158] In another example, a phase metric measures a speed performing a sequences of actions of the exercise action, e.g., pushing, pulling, flipping. Equation 8 below shows the average speed v of performing a sequence of actions i = 1 for n actions, where dLrepresents the displacement during action z and At, represents the duration of time for performing action i:

[0159] In another example, a phase metric measures an agility and precision of the athlete's footwork leading up to the exercise action. Equation 9 below shows an agility score A for an actual foot position p, relative to an ideal foot position p]deaZand a time between steps Atj for action i:,=i y tiip. - p ‘“'ih ™At; JUi=lx'

[0160] In another example, a phase metric measures a precision and steadiness of the athlete's eye alignment with the target, e.g., the strength training device. Equation 10 below shows an eye alignment error E for a gaze direction g(t) at time / , relative to a target or expected gaze direction gtarset(t) for a total duration TT^ = ^ / (llg(t) - gtar5et(t)||2) dt (10) o

[0161] In another example, a phase metric measures overall synchronization and coordination of all body parts during the single exercise action. Equation 11 below shows synchronization score S based on phase angles of body part i j (t) and a reference phase angle < I>re(t):

[0162] In another example, a phase metric measures a fidelity of the expected body part movements at an optimal time as compared to other body parts. Equation 12 below shows a fidelity score F for expected body movements, where Xi(tctual) represents actual position of body part i and Xi(t°ptimal) represents an expected position of body part i:

[0163] In another example, a phase metric quantifies an aggregate score of an athlete’s body stability and balance using the pose coordinates during the exercise action. Equation 13 below shows a balance score B based on a center of mass at time t shown as CoM(t) and a base of support BoS(t) for a total duration T:B = (13)0

[0164] In another example, a phase metric measures a movement and speed of the strength training device from when the athlete makes contact with the strength training device until the athlete releases the strength training device. For example, the position of the device can be dx ( t) represented by x(t), the velocity of the device can be represented by v(t) = and the,. i f dv (t)acceleration ot the device can be represented by a(t) =.

[0165] In some implementations, one or more phase metrics can be assessed in combination and / or in comparison to each other to generate global phase metrics. Phase metrics can also be generated based on evaluation of multiple action profiles in aggregate. For example, a phase metric can measure an accuracy of footwork, position, or the like, over an aggregate of multiple exercise actions taken by the athlete, e.g., during a workout program.

[0166] In some implementations, the strength training device form training system can leverage data collected by one or more sensors configured to capture non-visible light signal data, e.g., frequencies of the electromagnetic radiation spectrum outside the visible light spectrum, in addition to or instead of video and / or image data collected by an image sensor. For example, the system can leverage multi-modal sensor data collected by different types of sensors, e.g., at least one sensor configured to capture non-visible light signal data and at least one image sensor configured to capture video and / or image data. The multi-modal data can be enriched sensor data which includes additional insight, e.g., can include more information about the action than the video data alone.

[0167] FIG. 4 is a flow diagram of an example process 400 for the strength training device form training system. For convenience, the process 400 will be described as being performed by a system of one or more computers, located in one or more locations, and programmed appropriately in accordance with this specification. For example, a strength training device training system, e.g., the strength training device form training system 101 of FIG. 2, appropriately programmed, can perform the process 400.

[0168] The system evaluates (402), from the phase metrics of each phase of the one or more phases of the action profile, the action profile against an ideal action profile. As described above, the system can compare the action profile to an ideal action profile and / or self-consi stent action profile stored in database 204 for the particular exercise action and / or the set of exercise actions for the workout program. The comparison can include a comparison of timing and / or movement of each of the phases, where each phase is defined by the respective phase metrics. For example, a library of “ideal” or “exemplary” form data can be stored in computer memory. For each exercise action, a corresponding target form would specify the best form possible for that given exercise action. This can be defined in terms of form as described above, but also or instead in terms of phase and / or phase metric, depending on how the library is constructed and maintained.

[0169] With this comparison, the system can determine a measure of the quality of the exercise action according the different phases and corresponding phase metrics in addition to the make or miss of the exercise action. For example, an athlete may make a number of exercise actions in spite of the fact that they place their hands on the improper portion of the strength training device or take an idiosyncratic action before taking their exercise action. Similarly, an athlete may execute proper form on a high-difficulty exercise action that nevertheless fails. With this comparison, the system can determine an athlete’s process (i.e., their form) instead of their outcome. As will be appreciated, good form can be more predictive of future success in a particular drill than good outcomes - an athlete with good form who just is unable to complete high-difficulty exercise actions is more likely to progress than an athlete with inconsistent and wild form but who got lucky to complete an exercise action.

[0170] The system determines (404), from the evaluation, a deviation of the action profile from the ideal action profile for a phase of the one or more phases is larger than a threshold deviation. In some implementations, determining the deviation of the action profile from the ideal action profile for the phase includes determining a type of deviation of multiple types of deviations for the phase.

[0171] In some implementations, the system can determine an overall score for the exercise action. The system can determine the overall score as a combined score of each of the respective scores for the phases of the action profile. At times, the overall score can be a weighted value, e.g., where one phase is weighted more than another, different phase. At times, the overall score can be weighted, based in part on the exercise completion feedback, e.g., whether the exercise was completed and / or corrected. The overall score can be report, forexample, on a scale of 0 to 1 or 1 to 100 can include the various variances available upon request to help understand what contributed to the consistency score, e.g., broken down by phases and / or phase metrics.

[0172] In some implementations, the system uses the phase metrics to assess a consistency of the athlete’s exercise action. Consistency can be defined as how consistent an athlete is in summation (i.e., total exercise actions taken into consideration) as well as piecemeal (i.e., each phase of the exercise action). This will allow athletes to focus on areas in need of more attention. The system can measure consistency mathematically in various ways as described herein including the calculation of “standard deviation” of an athlete form over a series of exercise actions. Consistency can be calculated as a numerical representation (e.g., a consistency score) and depicted, in the graphical user interface, through charts and graphs, as well as a visualization of multiple exercise actions “overlayed” on top of each other to shows similarities and differences from one exercise action to another.

[0173] In addition to measuring form in a controlled environment (“static form” measurement), the system allows athletes to measure their dynamic form. In addition to measurement of dynamic form, the system can categorize exercise actions, e.g., automatically, semi-automatically, or by a user manually, with specific labels in order to allow for robust analysis of the athlete’s form in different contexts / dynamics. For example, form training can be applied to categories such as “push-pull,” “lift and flip.”

[0174] The system provides (406), to the person, a training alert. A training alert can be provided, by the system, visual display by way of a screen, audio display by way of a speaker, haptic display by way of a tactile element such a worn device with a vibration element, etc.

[0175] The strength training device form training system can provide the user with feedback that the athlete can incorporate while performing the workout. In one example, an athlete taking the low-probability exercise actions, e.g., additional weight added to the strength training device, that ends up as a failed exercise action. For this exercise action, the system can determine that the user’ s form is good in spite of failed exercise action. In response, the strength training device form training system can play a message through a speaker such as “Good form! You’ll get it soon.” Similarly, if the athlete then performs another exercise action with poor form, the real time feedback can tell the user what part of their form was off.

[0176] In some cases, this feedback and other feedback described in this document can be provided in a visual format that compares the recorded form to an ideal or exemplar form. In one instance, an exercise action with form representative of an athlete’s performance in a drillcan be presented as a video contemporaneous with the video of the exemplar. In one case, the athlete form, e.g., a skeletal representation or body form representation, can be overlaid in the video with partial transparency over video of a professional or expert athlete performing the same kind of exercise action. In one case, the athlete’s video can be shown next to the video of the expert.

[0177] In some implementations, providing the training alert by the system further includes determining, by the system and for the phase having a deviation larger than the threshold deviation, a training regimen specific to the phase and different than a training regimen for each other phase of the two or more phases, and providing the training alert. The system can determine the training regimen, e.g., workout, drills, etc., specific to the phase by determining the training regimen of a repository of training regimens that is specific to the type of deviation for the phase. For example, a first set of drills can be selected responsive to a timing deviation of the phase (e.g., too slow / fast, pace is off), a second set of drills can be selected responsive to a movement deviation of the phase (e.g., arms are in improper position relative to hips).

[0178] In some implementations, the system provides the training alert by providing, representation in a user interface, a visual representation indicating the phase of the one or more phases for which the deviation of the action profile from the ideal action profile is larger than the threshold deviation. In one example, the visual representation can include a listing of the different phases where each phase is color-coded, highlighted, or otherwise identified distinctly depending on the determined deviation of the action profile from the ideal action profile for the phase.

[0179] For example, as depicted in the graphical user interface 900 depicted FIG. 9, each of the phases, e.g., phases 902, 904, 906, displayed in the graphical user interface is coded with a different color / texture / shade to indicate “within threshold,” “borderline with respect to the threshold,” or “outside threshold,” e.g., using color indicators of green, yellow, and red. Additionally, each of the phases is displayed with a percentage value indicating a measure of alignment of the action profile with the ideal action profile, e.g., accuracy, consistency, etc.

[0180] In some implementations, the system provides, for presentation in the user interface, a selectable control operable to initiate the training regimen specific to correct a detected deviation for the phase.

[0181] In some implementations, the system further evaluates the action profile for the exercise action using one or more trained machine learning models. The system can use one or more trained machine learning models to receive, as input, exercise action data and provide, asoutput, a prediction of a form deviation for the action profile of the exercise action. For example, the system can provide exercise action data as input to one or more trained machine learning models, e.g., one model trained on lift data, one model trained on push data one model trained on follow-through data, etc. Each of the trained machine learning models can be trained using a respective, specific data set corresponding to collected data across multiple different athletes all performing the phase. As such, the trained machine learning models can be robust to different athlete sizes, variation in technique, deviations in video capture / resolution / camera position, and the like.

[0182] In some implementations, the machine learning models are trained using action data including or derived from multi-modal sensor data as input. For example, the training data can include video data and point cloud data of the action.

[0183] FIG. 5 is a flow diagram of an example process 500 for the strength training device form training system. For convenience, the process 500 will be described as being performed by a system of one or more computers, located in one or more locations, and programmed appropriately in accordance with this specification. For example, a strength training device training system, e.g., the strength training device form training system 101 of FIG. 2, appropriately programmed, can perform the process 500. The system provides (502), for each of the phases, a subset of the biometric tracking points tracking a timing and movement of the phase of the exercise action to a corresponding trained machine learning model. Alternatively, or additionally, the system can provide the sensor data capturing the phase of the exercise action to the trained machine learning model as input.

[0184] The system obtains (504) as output from each of the trained machine learning models phase metrics for the phase. The phase metrics, as described above, can include a prediction of deviation(s) of the user’s form from an ideal form and / or self-consistent form for the phase of the action profile. Phase metrics can include a prediction including a confidence score of the alignment of the user’s form for the phase with the ideal form and / or self-consi stent form.

[0185] In some implementations, the system collects sensor data capturing the user performing the exercise action two or more times, e.g., using one or more sensors, and aggregates the action profiles corresponding to respective exercise actions, e.g., in a sequence of actions performed during a workout program. This can allow the system to perform an aggregated analysis over multiple performed exercise actions and reduce any error introduced by accidental mistakes or transient issues.

[0186] FIG. 10 is a block diagram of computing devices 1000, 1050 that may be used to implement the systems and methods described in this document, as either a client or as a server or multiple servers. As an example, one or more components of the strength training device form training system can be an example of computing devices 1000, 1050. Computing device 1000 and 1050 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations described and / or claimed in this document.

[0187] Computing device 1000 includes a processor 1002, memory 1004, a storage device 1006, a high-speed interface 1008 connecting to memory 1004 and high-speed expansion ports 1010, and a low speed interface 1012 connecting to low speed bus 1014 and storage device 1006. Each of the components, e.g., processor 1002, memory 1004, storage device 1006, highspeed interface 1008, high-speed expansion ports 1010, and low speed interface 1012, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor 1002 can process instructions for execution within the computing device 1000, including instructions stored in the memory 1004 or on the storage device 1006 to display graphical information for a GUI on an external input / output device, such as display 1016 coupled to high speed interface 1008. In other implementations, multiple processors and / or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices 1000 may be connected, with each device providing portions of the necessary operations, e.g., as a server bank, a group of blade servers, or a multi-processor system.

[0188] The memory 1004 stores information within the computing device 1000. In one implementation, the memory 1004 is a computer-readable medium. In one implementation, the memory 1004 is a volatile memory unit or units. In another implementation, the memory 1004 is a non-volatile memory unit or units.

[0189] The storage device 1006 is capable of providing mass storage for the computing device 1000. In one implementation, the storage device 1006 is a computer-readable medium. In various different implementations, the storage device 1006 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configurations. In one implementation, a computer program product is tangibly embodied inan information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory 1004, the storage device 1006, or memory on processor 1002.

[0190] The high-speed controller 1008 manages bandwidth-intensive operations for the computing device 1000, while the low speed controller 1012 manages lower bandwidthintensive operations. Such allocation of duties is exemplary only. In one implementation, the high-speed controller 1008 is coupled to memory 1004, display 1016, e.g., through a graphics processor or accelerator, and to high-speed expansion ports 1010, which may accept various expansion cards (not shown). In the implementation, low-speed controller 1012 is coupled to storage device 1006 and low-speed expansion port 1014. The low-speed expansion port, which may include various communication ports, e.g., USB, Bluetooth, Ethernet, wireless Ethernet, may be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

[0191] The computing device 1000 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server 1020, or multiple times in a group of such servers. It may also be implemented as part of a rack server system 1024. In addition, it may be implemented in a personal computer such as a laptop computer 1022. Alternatively, components from computing device 1000 may be combined with other components in a mobile device (not shown), such as device 1050. Each of such devices may contain one or more of computing device 1000, 1050, and an entire system may be made up of multiple computing devices 1000, 1050 communicating with each other.

[0192] Computing device 1050 includes a processor 1052, memory 1064, an input / output device such as a display 1054, a communication interface 1066, and a transceiver 1068, among other components. The device 1050 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the components 1050, 1052, 1064, 1054, 1066, and 1068, are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other manners as appropriate.

[0193] The processor 1052 can process instructions for execution within the computing device 1050, including instructions stored in the memory 1064. The processor may also include separate analog and digital processors. The processor may provide, for example, for coordination of the other components of the device 1050, such as control of user interfaces, applications run by device 1050, and wireless communication by device 1050.

[0194] Processor 1052 may communicate with a user through control interface 1058 and display interface 1056 coupled to a display 1054. The display 1054 may be, for example, a TFT LCD display or an OLED display, or other appropriate display technology. The display interface 1056 may include appropriate circuitry for driving the display 1054 to present graphical and other information to a user. The control interface 1058 may receive commands from a user and convert them for submission to the processor 1052. In addition, an external interface 1062 may be provided in communication with processor 1052, so as to enable near area communication of device 1050 with other devices. External interface 1062 may provide, for example, for wired communication, e.g., via a docking procedure, or for wireless communication, e.g., via Bluetooth or other such technologies.

[0195] The memory 1064 stores information within the computing device 1050. In one implementation, the memory 1064 is a computer-readable medium. In one implementation, the memory 1064 is a volatile memory unit or units. In another implementation, the memory 1064 is a non-volatile memory unit or units. Expansion memory 1074 may also be provided and connected to device 1050 through expansion interface 1072, which may include, for example, a SIMM card interface. Such expansion memory 1074 may provide extra storage space for device 1050, or may also store applications or other information for device 1050. Specifically, expansion memory 1074 may include instructions to carry out or supplement the processes described above, and may include secure information also. Thus, for example, expansion memory 1074 may be provided as a security module for device 1050, and may be programmed with instructions that permit secure use of device 1050. In addition, secure applications may be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner.

[0196] The memory may include for example, flash memory and / or MRAM memory, as discussed below. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory 1064, expansion memory 1074, or memory on processor 1052.

[0197] Device 1050 may communicate wirelessly through communication interface 1066, which may include digital signal processing circuitry where necessary. Communication interface 1066 may provide for communications under various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA,CDMA2000, or GPRS, among others. Such communication may occur, for example, through radio-frequency transceiver 1068. In addition, short-range communication may occur, such as using a Bluetooth, WiFi, or other such transceiver (not shown). In addition, GPS receiver module 1070 may provide additional wireless data to device 1050, which may be used as appropriate by applications running on device 1050.

[0198] Device 1050 may also communicate audibly using audio codec 1060, which may receive spoken information from a user and convert it to usable digital information. Audio codec 1060 may likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of device 1050. Such sound may include sound from voice telephone calls, may include recorded sound, e.g., voice messages, music files, etc., and may also include sound generated by applications operating on device 1050.

[0199] The computing device 1050 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a cellular telephone 1080. It may also be implemented as part of a smartphone 1082, personal digital assistant, or other similar mobile device.

[0200] The subject matter and the actions and operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. The subject matter and the actions and operations described in this specification can be implemented as or in one or more computer programs, e.g., one or more modules of computer program instructions, encoded on a computer program carrier, for execution by, or to control the operation of, data processing apparatus. The carrier can be a tangible non-transitory computer storage medium. Alternatively or in addition, the carrier can be an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be or be part of a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. A computer storage medium is not a propagated signal.

[0201] The term “data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. Data processing apparatus can include specialpurpose logic circuitry, e.g., an FPGA (field programmable gate array), an ASIC(application-specific integrated circuit), or a GPU (graphics processing unit). The apparatus can also include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0202] A computer program can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program, e.g., as an app, or as a software module, component, engine, subroutine, or other unit suitable for executing in a computing environment, which environment may include one or more computers interconnected by a data communication network in one or more locations.

[0203] A computer program may, but need not, correspond to a file in a file system. A computer program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code.

[0204] The processes and logic flows described in this specification can be performed by one or more computers executing one or more computer programs to perform operations by operating on input data and generating output. The processes and logic flows can also be performed by special-purpose logic circuitry, e.g., an FPGA, an ASIC, or a GPU, or by a combination of special-purpose logic circuitry and one or more programmed computers.

[0205] Computers suitable for the execution of a computer program can be based on general or special-purpose microprocessors or microcontrollers or a combination of them, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special-purpose logic circuitry.

[0206] Generally, a computer will also include, or be operatively coupled to, one or more mass storage devices, and be configured to receive data from or transfer data to the mass storage devices. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video athlete, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0207] To provide for interaction with a user, the subject matter described in this specification can be implemented on one or more computers having, or configured to communicate with, a display device, e.g., a LCD (liquid crystal display) monitor, or a virtual -reality (VR) or augmented-reality (AR) display, for displaying information to the user, and an input device by which the user can provide input to the computer, e.g., a keyboard and a pointing device, e.g., a mouse, a trackball or touchpad. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback and responses provided to the user can be any form of sensory feedback, e.g., visual, auditory, speech, or tactile feedback or responses; and input from the user can be received in any form, including acoustic, speech, tactile, or eye tracking input, including touch motion or gestures, or kinetic motion or gestures or orientation motion or gestures. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser, or by interacting with an app running on a user device, e.g., on a smartphone or electronic tablet. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0208] This specification uses the term “configured to” in connection with systems, apparatus, and computer program components. That a system of one or more computers is configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. That one or more computer programs is configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions. That special-purpose logic circuitry is configured to perform particular operations or actions means that the circuitry has electronic logic that performs the operations or actions.

[0209] In addition to the embodiments described above, the following embodiments are also innovative:

[0210] Embodiment l is a form training system comprising: a strength training device; at least one sensor; and one or more processors coupled to one or more computer-readable storage media having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations comprising:obtaining, from the at least one sensor, sensor data including a person performing an exercise action with the strength training device;determining, from the sensor data, an exercise form comprising one or more points tracking the exercise action performed by the person, each point from the one or more points representing a location on the person;generating, by a model and for the exercise form, an evaluation indicating a deviation of the exercise form from a target form for performing the exercise action;determining, based on the evaluation for the exercise form, a form adjustment of the person for performing the exercise; andproviding, by a display device, data indicative of the form adjustment for output on the display device.

[0211] Embodiment 2 is the form training system of embodiment 1, wherein the strength training device is a football training sled.

[0212] Embodiment 3 is the form training system of embodiment 1, wherein the exercise comprises one or more of (i) flipping, (ii) lifting, or (iii) pushing, the strength training device.

[0213] Embodiment 4 is the form training system of embodiment 1, wherein the sensor data is captured by the at least one sensor prior to the exercise being performed.

[0214] Embodiment 5 is the form training system of embodiment 1, wherein the sensor data is captured by the at least one sensor after the exercise is performed by the person.

[0215] Embodiment 6 is the form training system of embodiment 1, wherein the model is configured to apply one or more of (i) machine learning techniques, (ii) statistical techniques, or (iii) a physical-based simulation, to generate the evaluation.

[0216] Embodiment 7 is the form training system of embodiment 1, wherein the evaluation comprises one or more of (i) a label, or (ii) a score, representing a deviation of the exercise form from the target form.

[0217] Embodiment 8 is the form training system of embodiment 1, the operations comprising:determining, based on the evaluation for the exercise form, a form adjustment of the person for performing the exercise; andproviding data indicative of the form adjustment for output.

[0218] Embodiment 9 is the form training system of embodiment 1, wherein providing data indicative of the form adjustment comprises providing the data for display on a user interface of a device communicatively coupled to the one or more processors, wherein the data causesthe device to configure the user interface to display one or more user interface elements to display the form adjustment.

[0219] Embodiment 10 is the form training system of embodiment 1, wherein the at least one sensor is mounted onto the strength training device.

[0220] Embodiment 11 is the form training system of embodiment 1, wherein the at least one sensor is located at a position remote from the strength training device.

[0221] Embodiment 12 is the form training system of embodiment 1, wherein the at least one sensor is co-located with the one or more processors.

[0222] Embodiment 13 is the form training system of embodiment 1, wherein the at least one sensor comprises is one or more of (i) an image sensor, (ii) a proximity sensor, (iii) an infrared sensor, or (iv) a radar sensor.

[0223] Embodiment 14 is the form training system of embodiment 1, the operations comprising:extracting, from the sensor data, action data comprising a plurality of points tracking the exercise action using the strength training device, wherein the plurality of points track respective locations of biometric data points corresponding to the person in the sensor data and tracking the exercise action;identifying, from the action data, two or more phases of the exercise action, wherein each of the two or more phases of the exercise action include a subset of the plurality of points tracking a timing and movement of the phase of the exercise action; andgenerating, from the action data, an exercise profile for the exercise action comprising the two or more phases.

[0224] Embodiment 15 is the form training system of embodiment 1, further comprising: evaluating the exercise profile for the exercise action, wherein the evaluating comprises:providing, for each of the two or more phases and to a corresponding trained machine learning model of a plurality of trained machine learning models, a subset of the plurality of pointstracking the timing and movement of the phase of the exercise action; and obtaining, for each of the two or more phases of the exercise action, phase metrics for the phase.

[0225] Embodiment 16 is the form training system of embodiment 1, wherein the plurality of trained machine learning models are each trained on a respective, different data set corresponding to a phase of the two or more phases of a exercise profile for the exercise action.

[0226] Embodiment 17 is the form training system of embodiment 1, further comprising: evaluating, from the phase metrics of each phase of the two or more phases of the exercise profile, the exercise profile against a target exercise profile;determining, from the evaluation, a deviation of the exercise profile from the target exercise profile for a phase of the two or more phases is larger than a threshold deviation; and providing, to the person, a training alert.

[0227] Embodiment 18 is the form training system of embodiment 17, wherein providing the training alert further comprise:determining, for the phase having a deviation larger than the threshold deviation, a training regimen specific to the phase and different than a training regimen for each other phase of the two or more phases; andproviding, to the person, the training alert.

[0228] Embodiment 19 is the form training system of embodiment 18, wherein determining a deviation of the exercise profile from the ideal exercise profile for the phase comprises determining, a type of deviation of a plurality of types of deviations for the phase, and wherein determining the training regimen specific to the phase comprises determining the training regimen from a plurality of training regimens that is specific to the type of deviation of the plurality of types of deviations for the phase.

[0229] Embodiment 20 is the form training system of embodiment 19, wherein providing the training alert further comprise:determining, for the phase having a deviation larger than the threshold deviation, a training regimen specific to the phase and different than a training regimen for each other phase of the two or more phases; andproviding, to the person, the training alert.

[0230] Embodiment 21 is the form training system of embodiment 20, wherein determining a deviation of the exercise profile from the ideal exercise profile for the phase comprises determining, a type of deviation of a plurality of types of deviations for the phase, and wherein determining the training regimen specific to the phase comprises determining the training regimen from a plurality of training regimens that is specific to the type of deviation of the plurality of types of deviations for the phase.

[0231] Embodiment 22 is the form training system of embodiment 20, wherein providing the training alert comprises:providing, for presentation in a user interface, a visual representation indicating the phase of the two or more phases for which the deviation of the exercise profile from the ideal exercise profile is larger than the threshold deviation; andproviding, for presentation in the user interface, a selectable control operable to initiate the training regimen specific to the phase.

[0232] Embodiment 23 is the form training system of embodiment 1, wherein obtaining the sensor data comprises obtaining the sensor data including the person performing the exercise action two or more times, andwherein generating an exercise profile comprises generating a respective exercise profile for each of the two or more performed exercise actions by the person.

[0233] Embodiment 24 is the form training system of embodiment 23, wherein obtaining the sensor data further comprises:identifying, for each of the respective exercise profiles, a corresponding reference point of the exercise profile; andaligning the exercise profiles for the two or more performed exercise actions using the respective reference points of the exercise profiles.

[0234] Embodiment 25 is the form training system of embodiment 24, further comprising:generating, from each of the two or more performed exercise actions, an average exercise profile for the exercise action for the person; andevaluating, from the average exercise profile against a target exercise profile; determining, from the evaluation, a deviation of the average exercise profile from the target exercise profile for a phase of the exercise action is larger than a threshold deviation; andproviding, to the person, a training alert.

[0235] Embodiment 26 is the form training system of embodiment 16, wherein the action data comprising the plurality of points tracking the exercise action further comprises tracking respective locations of an object captured by in the sensor data, andwherein tracking the respective locations of the object comprises tracking the object relative to the biometric data points corresponding to the person in the sensor data.

[0236] Embodiment 27 is the form training system of embodiment 16, wherein each phase of the two or more phases is defined by coordinate points and vectors for the points through a duration of the phase of the exercise action.

[0237] Embodiment 28 is the form training system of embodiment 16, where at least one phase of the two or more phases overlaps in timing with at least one other phase of the two or more phases during the exercise action.

[0238] Embodiment 29 is the form training system of embodiment 1, wherein the at least one sensor comprises an image sensor, andwherein the sensor data comprises video data including a plurality of frames including the person performing the exercise action.

[0239] Embodiment 30 is the form training system of embodiment 29, wherein obtaining, from the image sensor, the video data comprising the plurality of frames including the person performing the exercise action comprises:providing, for presentation in a user interface, a first visual indication of a first position to arrange the image sensor with respect to the person performing the exercise action to capture the video data;receiving, from the image sensor, the video data;determining, from the video data, that a threshold video data capturing the exercise action from the first position is met; andproviding, for presentation in the user interface, a second visual indication of a completion of capture of the video data from the first position.

[0240] Embodiment 31 is the form training system of embodiment 30, wherein obtaining, from the image sensor, the video data comprising the plurality of frames including the person performing the exercise action further comprises:providing, for presentation in a user interface, a third visual indication of a second position to arrange the image sensor with respect to the person performing the exercise action to capture the video data, wherein the second position is at least one of (i) a different position, or (ii) at a different angle, with respective to the form training system from the first position;receiving, from the image sensor, the video data;determining, from the video data, that a threshold video data capturing the exercise action from the second position is met; andproviding, for presentation in the user interface, a fourth visual indication of a completion of capture of the video data from the second position.

[0241] Embodiment 32 is the form training system of embodiment 30, wherein the first position is a side view of the person performing the exercise action, and wherein the second position is a face-on view of the person performing the exercise action.

[0242] Embodiment 33 is the form training system of embodiment 30, wherein determining, from the video data, that the threshold video data capturing the exercise action from the first position and the second position is met comprises:determining that a threshold number of exercise actions are captured in the plurality of frames of the video data from the respective first position and the second position.

[0243] Embodiment 34 is the form training system of embodiment 33, where the threshold number of exercise actions is different between the first position and the second position.

[0244] Embodiment 35 is the form training system of embodiment 30, wherein obtaining, from the at least one sensor, the sensor data including the person performing the exercise action further comprises:providing, to a trained machine learning model, the sensor data capturing the exercise action from the first position; andobtaining, from the trained machine learning model, extrapolated synthetic sensor data representing the exercise action from a second, virtual position.

[0245] Embodiment 36 is the form training system of embodiment 1, further comprising:obtaining, from the strength training device and for the exercise action, exercise action completion feedback; andgenerating, from an exercise profile and the exercise action completion feedback, an enriched exercise profile for the exercise action.

[0246] Embodiment 37 is the form training system of embodiment 1, wherein the at least one sensor is configured to capture sensor data in a non-visible electromagnetic spectrum frequency band.

[0247] Embodiment 38 is the form training system of embodiment 1, wherein the at least one sensor is configured to capture sensor data in one or more of a non-visible electromagnetic spectrum frequency band, radio frequency bands, microwave frequency bands, acoustic frequency bands, ultrasonic frequency bands, and infrared frequency bands.

[0248] Embodiment 39 is the form training system of embodiment 1, wherein the sensor data comprises point cloud data.

[0249] Embodiment 40 is the form training system of embodiment 1, wherein the at least one sensor is configured to capture sensor data in an ultrasonic frequency band.

[0250] Embodiment 41 is the form training system of embodiment 1, wherein the at least one sensor comprises a narrow-beam ultrasonic sensor.

[0251] Embodiment 42 is a form training method for a strength training device. The method comprisesproviding, by an emitter, a source signal comprising a non-visible wavelength; collecting, from a sensor and from reflections of the source signal, sensor data including a person performing an exercise action with a strength training device;extracting, from the sensor data, action data comprising a plurality of points tracking the exercise action, wherein the plurality of points track respective locations of biometric data points corresponding to the person appearing in the plurality of frames and tracking the exercise action;identifying, from the action data, one or more phases of the exercise action, wherein each of the one or more phases of the exercise action include a proper subset of the plurality of points tracking a timing and movement of the phase of the exercise action; and generating, from the action data, a movement profile for the exercise action comprising the one or more phases.

[0252] Embodiment 43 is a form training system comprising:a strength training device;at least one sensor; andone or more processors coupled to one or more computer-readable storage media having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations comprising:obtaining, from the at least one sensor, sensor data including a person performing an exercise action with the strength training device;determining, from the sensor data, an exercise form comprising one or more points tracking the exercise action performed by the person, each point from the one or more points representing a location on the person; andgenerating, based on the exercise form, a movement profile for the exercise action.

[0253] Embodiment 44 is the form training system of embodiment 43, wherein the strength training device is a football training sled.

[0254] Embodiment 45 is the form training system of embodiment 43, wherein the exercise comprises one or more of (i) flipping, (ii) lifting, or (iii) pushing, the strength training device.

[0255] Embodiment 46 is the form training system of embodiment 43, wherein the exercise comprises one or more motions performed by the person using the strength training device.

[0256] Embodiment 47 is the form training system of embodiment 43, wherein the sensor data is captured by the at least one sensor prior to the exercise being performed.

[0257] Embodiment 48 is the form training system of embodiment 43, wherein the sensor data is captured by the at least one sensor while the exercise is performed by the person.

[0258] Embodiment 49 is the form training system of embodiment 43, wherein the sensor data is captured by the at least one sensor after the exercise is performed by the person.

[0259] Embodiment 50 is the form training system of embodiment 43, wherein a model is configured to apply one or more of (i) machine learning techniques, (ii) statistical techniques, or (iii) a physical-based simulation, to generate an evaluation of the exercise form.

[0260] Embodiment 51 is the form training system of embodiment 43, wherein an evaluation of the exercise form comprises one or more of (i) a label, or (ii) a score, representing a deviation of the exercise form from a target exercise form.

[0261] Embodiment 52 is the form training system of embodiment 43, the operations comprising:determining, based on an evaluation for the exercise form, a form adjustment of the person for performing the exercise; andproviding data indicative of the form adjustment for output.

[0262] Embodiment 53 is the form training system of embodiment 52, wherein providing data indicative of the form adjustment comprises providing the data for display on a user interface of a device communicatively coupled to the one or more processors, wherein the data causes the device to configure the user interface to display one or more user interface elements to display the form adjustment.

[0263] Embodiment 54 is the form training system of embodiment 43, wherein the at least one sensor is mounted onto the strength training device.

[0264] Embodiment 55 is the form training system of embodiment 43, wherein the at least one sensor is located at a position remote from the strength training device.

[0265] Embodiment 56 is the form training system of embodiment 43, wherein the at least one sensor is co-located with the one or more processors.

[0266] Embodiment 57 is the form training system of embodiment 43, wherein the at least one sensor comprises is one or more of (i) an image sensor, (ii) a proximity sensor, (iii) an infrared sensor, or (iv) a radar sensor.

[0267] Embodiment 58 is the form training system of embodiment 43, the operations comprising:extracting, from the sensor data, action data comprising a plurality of points tracking the exercise action using the strength training device, wherein the plurality of points track respective locations of biometric data points corresponding to the person in the sensor data and tracking the exercise action;identifying, from the action data, one or more phases of the exercise action, wherein each of the one or more phases of the exercise action include a subset of the plurality of points tracking a timing and a movement of the phase of the exercise action; andgenerating, from the action data, an exercise profile for the exercise action comprising the one or more phases.

[0268] Embodiment 59 is the form training system of embodiment 58, further comprising evaluating the exercise profile for the exercise action, wherein the evaluating comprises: providing, for each of the one or more phases of the exercise action and to a corresponding trained machine learning model of a plurality of trained machine learning models, the subset of the plurality of points tracking the timing and the movement of the phase of the exercise action; andobtaining, for each of the one or more phases of the exercise action, phase metrics for the phase.

[0269] Embodiment 60 is the form training system of embodiment 59, wherein the plurality of trained machine learning models are each trained on a respective, different data set corresponding to a phase of the one or more phases of a exercise profile for the exercise action.

[0270] Embodiment 61 is the form training system of embodiment 60, further comprising:evaluating, from the phase metrics of each phase of the one or more phases of the exercise profile, the exercise profile against a target exercise profile;determining, from the evaluation, a deviation of the exercise profile from the target exercise profile for a phase of the exercise action is larger than a threshold deviation; and providing, to the person, a training alert.

[0271] Embodiment 62 is the form training system of embodiment 61, wherein providing the training alert further comprise:determining, for the phase having a deviation larger than the threshold deviation, a training regimen specific to the phase and different than a training regimen for each other phase of the one or more phases; andproviding, to the person, the training alert.

[0272] Embodiment 63 is the form training system of embodiment 62, wherein determining the deviation of the exercise profile from the ideal exercise profile for the phase comprises determining, a type of deviation of a plurality of types of deviations for the phase, and wherein determining the training regimen specific to the phase comprises determining the training regimen of a plurality of training regimens that is specific to the type of deviation of the plurality of types of deviations for the phase.

[0273] Embodiment 64 is the form training system of embodiment 62, wherein providing the training alert comprises:providing, for presentation in a user interface, a visual representation indicating the phase of the one or more phases for which the deviation of the exercise profile from the ideal exercise profile is larger than the threshold deviation; andproviding, for presentation in the user interface, a selectable control operable to initiate the training regimen specific to the phase.

[0274] Embodiment 65 is the form training system of embodiment 43, wherein obtaining the sensor data comprises obtaining the sensor data including the person performing the exercise action two or more times, andwherein generating an exercise profile comprises generating a respective exercise profile for each of the two or more performed exercise actions by the person.

[0275] Embodiment 66 is the form training system of embodiment 65, wherein obtaining the sensor data further comprises:identifying, for each of the respective exercise profiles, a corresponding reference point of the exercise profile; andaligning the exercise profiles for the two or more performed exercise actions using the respective reference points of the exercise profiles.

[0276] Embodiment 67 is the form training system of embodiment 65, further comprising:generating, from each of the two or more performed exercise actions, an average exercise profile for the exercise action for the person; andevaluating, from the average exercise profile against a target exercise profile; determining, from the evaluation, a deviation of the average exercise profile from the target exercise profile for a phase of the exercise action is larger than a threshold deviation; andproviding, to the person, a training alert.

[0277] Embodiment 68 is the form training system of embodiment 58, wherein the action data comprising the plurality of points tracking the exercise action further comprises tracking respective locations of an object captured by in the sensor data, andwherein tracking the respective locations of the object comprises tracking the object relative to the biometric data points corresponding to the person in the sensor data.

[0278] Embodiment 69 is the form training system of embodiment 58, wherein each phase of the one or more phases is defined by coordinate points and vectors for the points through a duration of the phase of the exercise action.

[0279] Embodiment 70 is the form training system of embodiment 58, wherein the action data comprises two or more phases of the exercise action and at least one phase of the two or more phases overlaps in timing with at least one other phase of the two or more phases during the exercise action.

[0280] Embodiment 71 is the form training system of embodiment 43, wherein the at least one sensor comprises an image sensor, andwherein the sensor data comprises video data including a plurality of frames including the person performing the exercise action.

[0281] Embodiment 72 is the form training system of embodiment 71, wherein obtaining, from the image sensor, the video data comprising the plurality of frames including the person performing the exercise action comprises:providing, for presentation in a user interface, a first visual indication of a first position to arrange the image sensor with respect to the person performing the exercise action to capture the video data;receiving, from the image sensor, the video data;determining, from the video data, that a threshold video data capturing the exercise action from the first position is met; andproviding, for presentation in the user interface, a second visual indication of a completion of capture of the video data from the first position.

[0282] Embodiment 73 is the form training system of embodiment 72, wherein obtaining, from the image sensor, the video data comprising the plurality of frames including the person performing the exercise action further comprises:providing, for presentation in the user interface, a third visual indication of a second position to arrange the image sensor with respect to the person performing the exercise actionto capture the video data, wherein the second position is at least one of (i) a different position, or (ii) at a different angle, with respective to the form training system from the first position;receiving, from the image sensor, the video data;determining, from the video data, that a threshold video data capturing the exercise action from the second position is met; andproviding, for presentation in the user interface, a fourth visual indication of a completion of capture of the video data from the second position.

[0283] Embodiment 74 is the form training system of embodiment 73, wherein the first position is a side view of the person performing the exercise action, and wherein the second position is a face-on view of the person performing the exercise action.

[0284] Embodiment 75 is the form training system of embodiment 73, wherein determining, from the video data, that the threshold video data capturing the exercise action from the first position and the second position is met comprises:determining that a threshold number of exercise actions are captured in the plurality of frames of the video data from the respective first position and the second position.

[0285] Embodiment 76 is the form training system of embodiment 75, where the threshold number of exercise actions is different between the first position and the second position.

[0286] Embodiment 77 is the form training system of embodiment 72, wherein obtaining, from the at least one sensor, the sensor data including the person performing the exercise action further comprises:providing, to a trained machine learning model, the sensor data capturing the exercise action from a first position; andobtaining, from the trained machine learning model, extrapolated synthetic sensor data representing the exercise action from a second, virtual position.

[0287] Embodiment 78 is the form training system of embodiment 43, further comprising:obtaining, from the strength training device and for the exercise action, exercise action completion feedback; andgenerating, from an exercise profile and the exercise action completion feedback, an enriched exercise profile for the exercise action.

[0288] Embodiment 79 is the form training system of embodiment 43, wherein the at least one sensor is configured to capture sensor data in a non-visible electromagnetic spectrum frequency band.

[0289] Embodiment 80 is the form training system of embodiment 43, wherein the at least one sensor is configured to capture sensor data in one of radio frequency bands, microwave frequency bands, acoustic frequency bands, and infrared frequency bands.

[0290] Embodiment 81 is the form training system of embodiment 43, wherein the sensor data comprises point cloud data.

[0291] Embodiment 82 is the form training system of embodiment 43, wherein the at least one sensor is configured to capture sensor data in an ultrasonic frequency band.

[0292] Embodiment 83 is the form training system of embodiment 43, wherein the at least one sensor comprises a narrow-beam ultrasonic sensor.

[0293] Embodiment 84 is a form training system comprising:a strength training device;at least one sensor; andone or more processors coupled to one or more computer-readable storage media having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations comprising:obtaining, from the at least one sensor, sensor data including a person performing an exercise action with the strength training device;determining, from the sensor data, an exercise form comprising one or more points tracking the exercise action performed by the person, each point from the one or more points representing a location on the person; andgenerating, by a model and for the exercise form, an evaluation indicating a deviation of the exercise form from a target form for performing the exercise action.

[0294] Embodiment 85 is the form training system of embodiment 84, wherein the at least one sensor is one of (i) mounted onto the strength training device, (ii) located at a position remote from the strength training device, and (iii) co-located with the one or more processors.

[0295] Embodiment 86 is the form training system of embodiment 84, wherein the sensor data is captured by the at least one sensor during a time period comprising at least one of (i) prior to the exercise action being performed by the person, (ii) while the exercise action is performed by the person, and (iii) after the exercise action is performed by the person.

[0296] Embodiment 87 is the form training system of embodiment 84, wherein the sensor data is captured by the at least one sensor prior to the exercise being performed.

[0297] Embodiment 88 is the form training system of embodiment 84, wherein the sensor data is captured by the at least one sensor while the exercise is performed by the person.

[0298] Embodiment 89 is the form training system of embodiment 84, wherein the sensor data is captured by the at least one sensor after the exercise is performed by the person.

[0299] Embodiment 90 is the form training system of embodiment 84, wherein the exercise action comprises one or more of (i) flipping, (ii) lifting, or (iii) pushing, the strength training device.

[0300] Embodiment 91 is the form training system of embodiment 84, wherein the exercise comprises one or more motions performed by the person using the strength training device.

[0301] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of what is being claimed, which is defined by the claims themselves, but rather as descriptions of features that may be specific to particular implementations of particular inventions. Certain features that are described in this specification in the context of separate implementations can also be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation can also be implemented in multiple implementations separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claim may be directed to a subcombination or variation of a subcombination.

[0302] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this by itself should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0303] Particular implementations of the subject matter have been described. Other implementations are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

What is claimed is:CLAIMS1. A form training system comprising:a strength training device;at least one sensor; andone or more processors coupled to one or more computer-readable storage media having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations comprising:obtaining, from the at least one sensor, sensor data including a person performing an exercise action with the strength training device;determining, from the sensor data, an exercise form comprising one or more points tracking the exercise action performed by the person, each point from the one or more points representing a location on the person;generating, by a model and for the exercise form, an evaluation indicating a deviation of the exercise form from a target form for performing the exercise action;determining, based on the evaluation for the exercise form, a form adjustment of the person for performing the exercise; andproviding, by a display device, data indicative of the form adjustment for output on the display device.

2. The system of claim 1, wherein the strength training device is a football training sled.

3. The system of claim 1, wherein the sensor data is captured by the at least one sensor during a time period comprising at least one of (i) prior to the exercise action being performed by the person, (ii) while the exercise action is performed by the person, and (iii) after the exercise action is performed by the person.

4. The system of claim 1, wherein the model is configured to apply one or more of (i) machine learning techniques, (ii) statistical techniques, or (iii) a physical-based simulation, to generate the evaluation.

5. The system of claim 1, wherein the evaluation comprises one or more of (i) a label, or (ii) a score, representing a deviation of the exercise form from the target form.

6. The system of claim 1, wherein providing data indicative of the form adjustment comprises providing the data for display on a user interface of a device communicatively coupled to the one or more processors, wherein the data causes the device to configure the user interface to display one or more user interface elements to display the form adjustment.

7. The system of claim 1, wherein the at least one sensor is one of (i) mounted onto the strength training device, (ii) located at a position remote from the strength training device, and (iii) co-located with the one or more processors.

8. The system of claim 1, wherein the at least one sensor comprises is one or more of (i) an image sensor, (ii) a proximity sensor, (iii) an infrared sensor, or (iv) a radar sensor.

9. The system of claim 1, the operations comprising:extracting, from the sensor data, action data comprising a plurality of points tracking the exercise action using the strength training device, wherein the plurality of points track respective locations of biometric data points corresponding to the person in the sensor data and tracking the exercise action;identifying, from the action data, two or more phases of the exercise action, wherein each of the two or more phases of the exercise action include a subset of the plurality of points tracking a timing and movement of the phase of the exercise action; andgenerating, from the action data, an exercise profile for the exercise action comprising the two or more phases.

10. The system of claim 9, further comprising evaluating the exercise profile for the exercise action, wherein the evaluating comprises:providing, for each of the two or more phases and to a corresponding trained machine learning model of a plurality of trained machine learning models, a subset ofthe plurality of points tracking the timing and movement of the phase of the exercise action; andobtaining, for each of the two or more phases of the exercise action, phase metrics for the phase.

11. The system of claim 9, wherein the action data comprising the plurality of points tracking the exercise action further comprises tracking respective locations of an object captured by in the sensor data, andwherein tracking the respective locations of the object comprises tracking the object relative to the biometric data points corresponding to the person in the sensor data.

12. The system of claim 9, wherein each phase of the two or more phases is defined by coordinate points and vectors for the points through a duration of the phase of the exercise action.

13. The system of claim 1, wherein the at least one sensor comprises an image sensor, and wherein the sensor data comprises video data including a plurality of frames including the person performing the exercise action.

14. The system of claim 13, wherein obtaining, from the image sensor, the video data comprising the plurality of frames including the person performing the exercise action comprises:providing, for presentation in a user interface, a first visual indication of a first position to arrange the image sensor with respect to the person performing the exercise action to capture the video data;receiving, from the image sensor, the video data;determining, from the video data, that a threshold video data capturing the exercise action from the first position is met; andproviding, for presentation in the user interface, a second visual indication of a completion of capture of the video data from the first position.

15. The system of claim 14, wherein obtaining, from the image sensor, the video data comprising the plurality of frames including the person performing the exercise action further comprises:providing, for presentation in the user interface, a third visual indication of a second position to arrange the image sensor with respect to the person performing the exercise action to capture the video data, wherein the second position is at least one of (i) a different position, or (ii) at a different angle, with respective to the form training system from the first position;receiving, from the image sensor, the video data;determining, from the video data, that a threshold video data capturing the exercise action from the second position is met; andproviding, for presentation in the user interface, a fourth visual indication of a completion of capture of the video data from the second position.

16. The system of claim 14, wherein obtaining, from the at least one sensor, the sensor data including the person performing the exercise action further comprises:providing, to a trained machine learning model, the sensor data capturing the exercise action from the first position; andobtaining, from the trained machine learning model, extrapolated synthetic sensor data representing the exercise action from a second, virtual position.

17. The system of claim 1, further comprising:obtaining, from the strength training device and for the exercise action, exercise action completion feedback; andgenerating, from an exercise profile and the exercise action completion feedback; an enriched exercise profile for the exercise action.

18. The system of claim 1, wherein the at least one sensor is configured to capture sensor data in one or more of a non-visible electromagnetic spectrum frequency band, radio frequency bands, microwave frequency bands, acoustic frequency bands, ultrasonic frequency bands, and infrared frequency bands.

19. A form training method for a strength training device, the method comprising: providing, by an emitter, a source signal comprising a non-visible wavelength; collecting, from a sensor and from reflections of the source signal, sensor data including a person performing an exercise action with a strength training device; extracting, from the sensor data, action data comprising a plurality of points tracking the exercise action, wherein the plurality of points track respective locations of biometric data points corresponding to the person appearing in the plurality of frames and tracking the exercise action;identifying, from the action data, one or more phases of the exercise action, wherein each of the one or more phases of the exercise action include a proper subset of the plurality of points tracking a timing and movement of the phase of the exercise action; andgenerating, from the action data, a movement profile for the exercise action comprising the one or more phases.

20. A form training system comprising:a strength training device;at least one sensor; andone or more processors coupled to one or more computer-readable storage media having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to perform operations comprising:obtaining, from the at least one sensor, sensor data including a person performing an exercise action with the strength training device;determining, from the sensor data, an exercise form comprising one or more points tracking the exercise action performed by the person, each point from the one or more points representing a location on the person; andgenerating, based on the exercise form, a movement profile for the exercise action.

Images