Transmissionless gripper device for robot model training

The transmissionless gripper device with rotary joints and integrated sensors addresses the challenges of data collection for robotic manipulation by enabling intuitive control and precise tracking, enhancing the efficiency and accuracy of robot training data, thus improving robotic manipulation tasks.

WO2026236108A1PCT designated stage Publication Date: 2026-11-12GENERALIST AI INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/US2026/026514
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-05
Filing Date
2026-05-04
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

Existing robotic manipulation systems face challenges in efficiently collecting high-quality training data for machine learning models due to mechanical complexity, lack of intuitive control, and inadequate haptic feedback, which affects the precision and naturalness of robot manipulation tasks.

Method used

A transmissionless gripper device with rotary joints and integrated sensors, including a camera assembly and visual markers, allows users to intuitively control finger movement, providing haptic feedback and precise tracking, while generating synchronized image and position data for training computational models to replicate human manipulation behaviors.

Benefits of technology

The gripper device enables efficient collection and processing of high-quality training data, allowing computational models to accurately control robots to perform various manipulation tasks with improved precision and naturalness, reducing user fatigue and mechanical complexity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure US2026026514_12112026_PF_FP_ABST
    Figure US2026026514_12112026_PF_FP_ABST
Patent Text Reader

Abstract

Examples relate to a system for generating robot training data. A gripper device mountable to a user's hand includes a first finger rotatable inward by thumb force and a second finger rotatable inward by finger force, with a camera assembly mounted to capture image data of the fingers and workspace. The transmissionless design enables direct actuation by the user's digits, providing intuitive control and haptic feedback. The camera assembly captures both the manipulation workspace and surrounding environment to enable robust visual tracking. The device features integrated visual markers and compliant regions that provide grip position and force feedback without requiring sensors. The captured image data is used to train computational models that control robot manipulators having matching visual and mechanical properties.
Need to check novelty before this filing date? Find Prior Art

Description

TRANSMISSIONLESS GRIPPER DEVICE FOR ROBOT MODEL TRAININGPRIORITY

[0001] This application claims the benefit of priority to US. Patent Application Serial No. 19 / 198,481. filed May 5. 2025, which is incorporated by reference herein in its entirety.TECHNICAL FIELD

[0002] The present disclosures relate to robotic control systems and, in some examples, to devices and systems to generate training data using hand-mountable gripper devices with integrated sensors for training a robot control model to perform manipulation tasks.BACKGROUND

[0003] Robotic manipulation systems use cameras, sensors, and mechanical grippers to enable robots to interact with objects in their environment. These systems may employ computational models or algorithms configured to combine computer vision for perceiving the environment, motion planning for determining movement trajectories, and control schemes for executing precise manipulations. Some systems use machine learning techniques to train these models using training data. Training data collection may involve recording demonstrations of manipulation tasks, which can then be used to train machine learning models to teach the model to control robots to replicate similar behaviors.BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

[0004] Example embodiments are described in detail below with reference to the attached drawing figures.

[0005] FIG. 1 is a perspective view illustrating an example gripper device, according to some examples.

[0006] FIG. 2 is a right side view of a user's hand wearing the gripper device of FIG. 1 with field of view indicators, according to some examples.

[0007] FIG. 3 is a perspective view illustrating an alternative example gripper device with visually compliant regions and contrasting markers, according to some examples.

[0008] FIG. 4 is an exploded view illustrating components of the gripper device of FIG. 3. according to some examples.

[0009] FIG. 5 is a top view of a robot manipulator corresponding to the gripper device of FIG. 3, according to some examples.

[0010] FIG. 6 is a front view of a robot system with two arms equipped with the robot manipulators of FIG. 5, according to some examples.

[0011] FIG. 7 is a top view of different fingertip designs for a gripper device and robot manipulator with various grip textures and patterns, according to some examples.

[0012] FIG. 8 is a perspective view of fingertips of a gripper device or robot manipulator, according to some examples.

[0013] FIG. 9 is a perspective view of a visually compliant region of a fingertip of a gripper device or robot manipulator showing honeycomb structures deforming in response to force from engaging a surface, according to some examples.

[0014] FIG. 10 illustrates different grip surface textures and patterns, according to some examples.

[0015] FIG. 11 is a block diagram illustrating a system for training a computational model to operate a robot using data from gripper devices, according to some examples.

[0016] FIG. 12 is a flowchart illustrating operations of a method for training a computational model to operate a robot using data from gripper devices, according to some examples.

[0017] FIG. 13 illustrates a machine-learning pipeline, according to some examples.

[0018] FIG. 14 illustrates training and use of a machine-learning program, according to some examples.DETAILED DESCRIPTION

[0019] Described examples provide systems and methods for collecting human demonstration data to train robots to perform manipulation tasks. A gripper device mounts to a user's hand and includes a first finger rotatable inward by the thumb and a second finger rotatable inward by other fingers. The device employs a transmissionless design in which the user directly actuates the fingers through rotary joints rather than linear rails or trigger mechanisms, enabling intuitive control and haptic feedback. A common rotational mechanism constrains the fingers to rotate jointly using gears, simplifying the grip width to a one-dimensional value.

[0020] A camera assembly mounted on the device captures both the manipulation workspace (including the device's fingers) and the environmental context. In some examples, a wide-angle or 360-degree camera provides an expanded field of view that includes features useful for spatial tracking, such as comers of walls, floors, and / or ceilings of an indoor environment. In some examples, the camera's center of mass is positioned above the user's hand to improve ergonomics. A mask can occlude the user's fingers from the camera view to better match the robot's perspective when employing a corresponding robot manipulator.

[0021] In some examples, the fingers incorporate specialized features including visually compliant regions with honeycomb structures that deform to provide visual indication of applied forces without requiring force sensors. High-contrast visual markers can be integrally molded into or otherwise integrally formed with the finger structure rather than added afterward, enabling precise tracking of finger positions. The fingers can also include multi-stiffness zones, multi-antipodal corresponding gripping surfaces, and textured gripping surfaces optimized for different manipulation tasks. Other mechanical gripping features and ergonomic features are described herein to improve the intuitiveness, effectiveness, and usability of the gripper devices to enable more effective collection of training data.

[0022] The system includes robot manipulators designed to match the visual and mechanical properties of the gripper devices. In some examples, the manipulators use low-backlash metal gears to precisely control finger positions. The robot manipulator fingers can be manufactured using the same injection molding process as the gripper devices to ensure consistent appearance and behavior. The hardware-agnostic design of the robot manipulators supports deployment across different robot platforms.

[0023] In some examples, the gripper device includes position sensors, such as an inertial measurement unit (IMU) integrated into the camera assembly, to generate position information (e.g., six-dimensional x-y-z-pitch-roll-yaw information) synchronized with the image data captured by the camera assembly. The position information and image data can be processed to generate simultaneous localization and mapping (SLAM) data for use in training the model. Additional sensing capabilities are described herein to improve the value and usability7of the collected training data.

[0024] The mechanical, visual, and sensor features of the gripper devices and corresponding robot manipulators of the system enable the efficient collection, processing, sorting, categorization, and application of high-quality training data to a machine learning pipeline for training computational models to control robots to perform various manipulation tasks. In some examples, the video and / or other sensor modality data is collected from a set of gripper devices while a user performs a manipulation task, then is automatically paired and synchronized to match left and right counterpart gripper data generated during the same manipulation task, is processed to generate secondary- data streams (e g., SLAM data, grip width data, visual deformation data indicative of forces acting on the fingers), and is provided as training data inputs to a model training process.Computational models are trained to control robots to replicate demonstrated manipulation behaviors while adapting to variations in the environment. The complete system integrates the gripper devices, robot manipulators, and computational pipeline to enable demonstration-based robot training across various robot platforms.

[0025] FIG. 1 illustrates a perspective view of an example gripper device 100.

[0026] The gripper device 100 includes a camera assembly 106 mounted above and at a rear portion the device, close to the user's wrist. A left finger 102 and a right finger 104 extend forward from a common rotational mechanism 108 (shown in greater detail in FIG. 4). The fingers are rotatable inward toward each other through rotational joints 110.

[0027] In some examples, each finger includes a gripping member 112 with an inner surface 114 for gripping objects. The gripping member 112 extends inward from a main body 128 of the finger toward an opposite inner surface 114 of the other finger. A rear end of the main body 128 is coupled to the rotational joint 110, and the main body 128 extends forward to form an outer surface 116 to accommodate the user's fingers or thumb. The fingers or thumb are occluded or obscured from the field of view of the camera assembly 106 by a mask 120 formed from the main body 128 and extending outward above the outer surface 116. The mask 120 incorporates visual tracking features 118 integrally formed into the structure of the main body 128. The visual tracking features 118 present high-contrast fiducial reference points for tracking finger positions and grip width using image data captured by the camera assembly 106. The visual tracking features 118 are arranged in patterns along the fingers, and can be positioned at key points to assist in a visual determination of a grip width or radial position of the fingers, because both fingers are constrained to the same magnitude of radial displacement from a baseline position.

[0028] The transmissionless design allows direct actuation of the fingers by the user's thumb and fingers without intermediate mechanisms. In some examples, the common rotational mechanism 108 constrains the fingers to rotate jointly using gears coupled to the rotational joints 110, simplifying the measurement of grip width to a one-dimensional value.

[0029] In some examples, the gripper device 100 incorporates an offset angle (e.g., 20 degrees) between the human hand axis and the gripper finger axis to improve ergonomics during tabletop manipulation tasks. Such anangled ‘‘pencil grip” design can reduce wrist strain by allowing more natural hand positioning while maintaining precise control of the gripper fingers. This ergonomic feature can enable extended use during data collection with reduced user fatigue.

[0030] The transmissionless, rotary joint-based design may reduce mechanical complexity and improve the naturalness and effectiveness of manipulation compared to linear actuation systems. Trigger-based linear actuation, in which a linear actuator squeezes gripping members toward each other, not only displaces the mechanism forward from the user's hand, making manipulation more awkward and less intuitive, but it also provides less precise grip control and haptic feedback to the user. In contrast, the glove-like transmissionless design of the gripper device 100 allows a user to grip objects naturally at a very' short distance from the user's hand using motions and relying on haptic feedback that closely emulate natural hand gripping tasks.

[0031] In addition, the rotational joints 110 enable a natural and versatile gripping style, in contrast to gripper designs using a pair of parallel gripping members mounted on a linear rail. Not only such a linear parallel gripper more unnatural and less versatile for performing manipulation, but the linear rail presents a wide obstruction of a fixed width at the base on the parallel gripping members, preventing the gripping members from navigating tight spaces (such as insertion deep into a tight cavity) even when fully closed. In contrast, the rotational design of the gripper device 100 allows the fingers to be rotated between a very7wide open position suitable for grasping large objects and a very narrow closed position that presents a narrow forwardfacing profile.

[0032] In some examples, the rotational movement of the fingers is leveraged to provide a multi-antipodal pair of gripping surfaces on the inner surfaces 114 of the gripping members 112. As shown in FIG. 1, each inner surface 114 defines multiple rounded concavities; when the fingers rotate inward and outward, these complementary7pairs of concavities face each other at different angles of rotation, defining different antipodal pairs ofpoints on the two surfaces where the surface normal vectors of the two inner surfaces 114 are colinear, enabling an object to be gripped between the complementary antipodal points. For example, a first pair of antipodal points 122 of the largest, rear-most concavity face each other when the fingers are rotated fully inward to a closed position, whereas a second pair of antipodal points 124 face each other when the fingers are rotated fully outward to a fully open position. In the illustrated example, the gripping members 112 include additional, smaller concavities located farther forward, each of which includes multiple such pairs of antipodal points that face each other at different grip widths.

[0033] In some examples, the inner surfaces 114 of the gripping members 112 are textured to improve frictional engagement with gripped objects. The gripping members 112 can also be formed from a deformable material, and in some cases may include multiple regions each having a different degree of stiffness or flexibility. When combined with the multi-antipodal pair of gripping surfaces described above, the textured inner surfaces 114 and / or deformable properties of the gripping members 112 can further enhance the frictional engagement with gripped objects as the fingers are rotated toward each other, compressing the deformable material and / or deforming texture protrusions to increase the force and surface area of the gripping members 112 engaging with surfaces of the object. Details of possible textures for the inner surfaces 114 are described below with reference to FIG. 10.

[0034] The gripper device 100 may incorporate or exhibit one or more ergonomic features or properties to improve the comfort, ease, and naturalness of manipulations tasks performed by the user's hand. The camera assembly 106 can be positioned such that its center of mass is close to or slightly behind the user's wrist (e g., directly above a rear portion of the gripper device 100, as show n in FIG. 1). This situation of the camera assembly 106 can provide a counterw eight to the fingers and other components of the gripper device 100 to reduce torque applied to the user's hand and / or to reduce the moment of inertia of the gripper device 100, thereby providing a more comfortable and natural balance to the user's hand.A biasing mechanism (not shown) can be included, for example as part of the common rotational mechanism 108, to bias the fingers toward an open position, thus reducing the need for the user's fingers and thumb to actively pull the fingers outward when opening the device's grip. Straps (as shown below in FIG. 2) can be included to couple the fingers to the user's thumb and fingers, and / or to couple the device as a whole to the user's hand or wrist, reducing the need for the user to maintain a grip on the gripper device 100 to prevent it from falling or slipping, and maintaining a firm engagement of the thumb and fingers with the outer surfaces 116. The outer surfaces 116 can include a deformable material, such as a foam layer, to absorb sweat and improve the comfort and reliability of the engagement of the outer surfaces 116 with the thumb and fingers.

[0035] In some examples, the fingers incorporate visually compliant regions that deform to provide visual indication of applied forces without requiring force sensors. For example, a deformable honeycomb structure can be formed into an upper surface] 126 of the gripping members 112 to be visible by the camera assembly 106. In some examples, a visually compliant region can be included near the fingertips of the fingers to show forces acting near the fingertips. In some examples, a visually compliant region can be included in the upper surface] 126 near the inner surface 114 to most effectively show forces acting on the inner surfaces 114. In some examples, the fingers can include regions of varying stiffness, as described above; these regions can include a fingernail structure incorporated into the fingertip to enable hard engagement with objects at the fingertip while maintaining flexibility7in the finger as a whole. Details of various fingertip designs and visually compliant regions are described below with reference to FIG. 7 through FIG. 9.

[0036] The overall design enables intuitive control and haptic feedback while providing visual features necessary for training computational models to control robot manipulators. In some examples, the mechanical components are designed for cost-effective and highly uniform mass production, such as through injection molding. In some examples, the mainbody 128 of each finger and its corresponding gripping member 112 can be integrally formed together from two different materials (e.g.. a more compliant or deformable material for the gripping member 112) via a two-shot injection molding process. Other components, such as brackets and plates for mounting the camera assembly 106 and common rotational mechanism 108, can also be injection molded (e.g., from the same material as the main body 128 of the fingers). Examples of the form and assembly of the various components of the gripper device 100 are described below with reference to FIG. 4.

[0037] It will be appreciated that the design of the gripper device 100 is ambidextrous, such that the same gripper device 100 can be used in the left hand or right hand. This means that only a single type of device needs to be manufactured, and any one device can be used in place of another. It eliminates the need for left and right hand devices to be paired together and simplifies replacement or substitution of faulty or unavailable devices. In some examples, a collection of gripper devices 100 can be supplied to a data collection team, and any user performing a given manipulation task can use any two of the devices to collect the data. Example systems and methods for data collection and model training, described below with reference to FIG.11 and FIG. 12, can include a step of automatically pairing and synchronizing data from two devices used in a user's left and right hands to perform the same manipulation task. This pairing can be performed using portions of the data from the two devices, such as synchronized audio cues in audio data captured by a microphone the camera assembly 106, visual cues from the environment embedded in the image data, time stamp data, location data, and so on.

[0038] The mechanical features of the gripper device 100 can interact with its sensing features to generate highly detailed image and position data encoding rich and precise information about the position of the device, the grip width of the fingers, the forces exerted by objects engaged by the fingers, the appearance of the objects being manipulated, and position of other objects (such as working surfaces) in the workspace. This detailedimage and position data can be used to train a model to control a robot to perform the same manipulation tasks using robot manipulators that present a highly similar visual profile to a similarly positioned and configured camera assembly.

[0039] In some examples, the gripper device 100 includes markers, such as April grid markers, QR codes, or printed numbers, that are visible to the camera assembly 106 and that encode various metadata including operator identifier codes, collection site identification, task information, and / or hardware version tracking. The encoded information enables automated tracking and organization of data collected from multiple devices and locations.

[0040] Thus, in some examples, a system for training a robot 600 includes a gripper device 100 mountable to a user's hand 202. The gripper device 100 includes a first finger (e g., left finger 102) rotatable inward by a thumb and a second finger (e.g., right finger 104) rotatable inward by other fingers of the hand 202. A camera assembly 106 mounted to the gripper device captures image data over a field of view that includes portions of the fingers and a workspace 212 extending forward from the fingers.

[0041] In some examples, the gripper device includes position sensors to generate position data representative of a position of the gripper device 100. The position data may include six-dimensional x-y-z-pitch-roll-yaw information synchronized with image data from the camera assembly.

[0042] In some examples, the gripper device 100 includes a common rotational mechanism 108 with gears (shown in FIG. 4 below) that constrain the left finger 102 and right finger 104 to rotate jointly, such that radial positions of both fingers are defined by a one-dimensional grip index. The gears couple to rotational joints 110 to ensure synchronized movement of the fingers.

[0043] In some examples, visual tracking features 118 are integrally formed with the fingers through a two-shot injection molding process to present high-contrast fiducial reference points in the field of view of the camera assembly 106. The visual tracking features 118 are arranged in patternsalong the fingers to enable precise tracking of finger positions and grip width.

[0044] In some examples, the first finger and second finger mirror each other's structure to permit use of the gripper device 100 on either hand, with the first finger becoming the right finger 104 when worn on the left hand and the left finger 102 when worn on the right hand. The mirrored structure enables ambidextrous use while maintaining consistent visual and mechanical properties.

[0045] In some examples, the structure of the left finger 102 and right finger 104 are formed by injection molding processes that enable cost-effective mass production while ensuring consistent quality and precise dimensional control needed for reliable operation.

[0046] In some examples, the camera assembly 106 is positioned with its center of mass above the user's hand to improve ergonomics by minimizing torque applied to the hand. The positioning of the camera assembly 106 provides a counterweight to the fingers and other components.

[0047] In some examples, the gripper device 100 includes a mask 120 to occlude the user's fingers from the field of view of the camera assembly 106. The mask 120 helps match the visual profile of the gripper device to that of a corresponding robot manipulator.

[0048] In some examples, the fingers each include a deformable gripping member 112 with an inner surface 114 having multiple stiffness zones, including a fingernail structure at the tip. The inner surfaces define multi-antipodal gripping surfaces that correspond at different grip widths, with textured surfaces for improved grip.

[0049] FIG. 2 illustrates a right side view of a user's hand 202 wearing the gripper device 100, showing field of view indicators.

[0050] The gripper device 100 includes the camera assembly 106 mounted above the user's hand 202. The camera assembly 106 captures image data over a field of view which can span various angular ranges. For example, the camera assembly 106 can provide a narrow field of view 208 that shows only a portion of the left finger 102 (not visible in FIG. 2) and right finger 104 ofthe gripper device 100, along with at least a portion of a workspace 212 in which the manipulation task is being performed. In the illustrated example, the workspace 212 extends forward from the tips of the fingers of the gripper device 100, and can encompass objects being manipulated as well as the immediate environment of those objects, such as nearby obstructions as well as work surfaces on which the objects rest. The narrow field of view 208 may be configured to include the visual tracking features 118 of the gripper device 100, as well as any visually compliant regions of the device's fingers, in order to improve the usefulness of the image data generated by the camera assembly 106.

[0051] In some examples, the camera assembly 106 provides a wider field of view, such as a 360 degree field of view (e.g., using a 360 degree camera) or a wide field of view 210 (e.g., using a wide-angle lens). In some examples, the camera assembly 106 includes multiple cameras, each having its own field of view, and the field of view of the camera assembly 106 is a combined field of view including the field of view of each camera.

[0052] In some examples, the field of view is wide enough to encompass not only the left finger 102, right finger 104, and workspace 212, but also at least a portion of the wider environment 216 around the gripper device 100. For example, the wide field of view 210 can be wide enough to encompass features 218 angularly displaced by more than 90 degrees from an axis 214 extending forward from the gripper device 100. This wide field of view 210 enables the camera assembly 106 to track visual features of the environment 216 such as the ceiling, floor, walls, and other reference surfaces, corners, and points that can assist with SLAM and other visual navigation and mapping operations. In some examples, the camera assembly 106 can be used to capture a very wide field of view (such as 360 degrees in horizontal and vertical planes), and the resulting image data can be preprocessed to mask or filter out irrelevant distractions, such as humans or other moving objects visible in the image data. The image data of the region directly around the gripper device 100 can present a visual profile of the device and the manipulation task that is nearly identical to the view of a correspondingrobot manipulator (described in greater detail below with reference to FIG. 5 and FIG. 6), in part due to the mask 120 used to occlude the user's thumb and fingers holding the gripper device 100.

[0053] Thus, in some examples, the field of view of the camera assembly 106 includes portions of the environment 216 outside the workspace 212, including features 218 displaced more than 90 degrees from an axis 214 extending forward from the gripper device 100. This expanded field of view enables tracking of environmental features useful for spatial tracking and registration.

[0054] In some examples, as described above, straps 206 can be included in the gripper device 100 to secure the gripper device 100 to the user's hand 202, thumb, and / or one or more fingers 204. The straps 206 may include a strap to secure the device's right finger 104 to one or more of the user's fingers 204, a strap (not visible) to secure the left finger 102 (not visible) to the user's thumb, and / or a strap 206 to secure the gripper device 100 to the wrist or palm of the user's hand.

[0055] In some examples, the straps 206 include a strap to secure the first finger of the gripper device 100 to the hand 202, a strap to secure the second finger of the gripper device 100 to one or more fingers of the hand 202, and a strap to secure the gripper device to the wrist or palm of the hand 202. The multiple straps ensure secure mounting while maintaining natural movement.

[0056] Thus, in some examples, the gripper device includes a biasing mechanism to bias the first finger and second finger away from each other, and straps 206 to secure the device to the user's hand 202. The biasing mechanism reduces user fatigue while the straps maintain secure engagement.

[0057] FIG. 3 illustrates a perspective view of an alternative gripper device 300 having many of the same features as the gripper device 100 of FIG. 1, similarly numbered in FIG. 3.

[0058] The gripper device 300 features specialized fingertips 302. In some examples, the fingertips 302 are formed from a stiffer material than thegripping members 112 and define visually compliant regions that visibly deform in response to forces acting on the fingertips 302.

[0059] It will be appreciated that the overall design of the gripper device can be varied while still exhibiting many or all of the features described above with reference to the gripper device 100. However, it is important for any design for the gripper device (e.g., gripper device 100 or gripper device 300) to visually and mechanically correspond to its robot manipulator counterpart in order to ensure reliable and accurate control of the robot based on the image data captured by the robot's cameras.

[0060] FIG. 4 illustrates an exploded view of components of the gripper device 300. It will be appreciated that gripper device 100 includes similar components, similarly assembled.

[0061] The gripper device 300 is formed from a modular assembly of injection-molded components, including a camera bracket 406 for mounting the camera assembly 106, a base plate 410 for assembling the common rotational mechanism 108, and the fingers (left finger 102 and right finger 104). As described above, in some cases the fingers are formed by a two-shot injection molding process, with a first material used for the main body 128 and a second material used for the gripping member. In some examples, the base plate 410 and / or camera bracket 406 can be injection molded from the same material as the main body 128 of the fingers.

[0062] In some examples, the common rotational mechanism 108 includes metal gears 402 that constrain the fingers to rotate jointly. The gears 402 are coupled to the rotational joints 110 to ensure synchronized movement of the fingers. Multiple spacers 404 are positioned between components to maintain proper alignment and spacing. In some examples, the spacers 404 are machined from metal. The common rotational mechanism 108 can include a biasing mechanism (not visible), such as a spring, engaging the gears 402 or another component coupled to the fingers to bias the fingers to an open position.

[0063] The assembly of the gripper device 300 includes foam 408 providing padding on the outer surfaces 116 of the fingers to contact theuser's hand to improve comfort and grip. The foam 408 may be die-cut to precisely fit the contours of the outer surfaces 116 of the fingers.

[0064] The camera assembly 106 mounts to the camera bracket 406, which can be designed to position the camera's center of mass above the user's hand. The base plate 410 provides structural support and mounting points for the other components.

[0065] In some examples, the two-shot injection molding process used to form the fingers (102, 104) also integrally forms the visual tracking features 118 and visually compliant regions of the fingers using visually contrasting materials, such as yellow for the tracking features and gripping members 112, and black for the main body 128 of the fingers (including the mask 120). This manufacturing approach can eliminate the need for separately applied markers or grip surfaces.

[0066] The modular design enables cost-effective mass production and easy assembly / maintenance. The use of injection molding for key components ensures consistent quality and precise dimensional control needed for reliable operation of the gearing mechanism and visual tracking features.

[0067] FIG. 5 illustrates a top view of a robot manipulator 500 corresponding to the gripper device 300 of FIG. 3.

[0068] The robot manipulator 500 conforms closely to the mechanical behavior, structural form, and visual profile of the gripper device 300, thereby enabling a model training using data from gripper devices 300 to be used to control a robot equipped with robot manipulators 500 relying on image data from a camera assembly 106 mounted to each robot manipulator 500.

[0069] The robot manipulator 500 includes an actuator 502 for actuation by the robot. In some examples, the actuator 502 includes low-backlash gears 504 that constrain rotation of the fingers (102, 104) to joint rotation, like the common rotational mechanism 108 of the gripper device 100 or gripper device 300. However, the use of low-backlash gears 504 in the actuator 502 enables precise control of finger positions while maintaining the same one-dimensional grip width constraint as the common rotational mechanism 108 of the gripper device 300.

[0070] The fingers (102, 104) can be manufactured using the same injection molding process as the fingers of the gripper device 300 to ensure consistent appearance and behavior. The fingers incorporate visual tracking features 118 and a mask 120 that are visually identical to those of the gripper device 300 to maintain a matching visual profile when viewed by the robot's camera system.

[0071] In some examples, the robot manipulator 500 is designed to be hardware-agnostic, enabling deployment across different robot platforms while maintaining the same mechanical and visual properties that correspond to the gripper device 300. This correspondence between the human-operated gripper device and robot manipulator enables reliable transfer of demonstrated manipulation behaviors to the robot system.

[0072] Thus, in some examples, a robot manipulator 500 includes a pair of robot manipulator fingers constrained by low-backlash gears 504, a camera assembly matching the gripper device 100, and visual tracking features integrally formed in the fingers. The robot manipulator 500 matches the mechanical and visual properties of the gripper device to enable reliable transfer of demonstrated behaviors.

[0073] FIG. 6 illustrates a front view of a robot 600 with two arms equipped with robot manipulators, such as the robot manipulators 500 shown in FIG. 5.

[0074] The robot 600 includes a first arm 602 and a second arm 604 mounted to a support structure. A first robot manipulator 606 (e.g., a robot manipulator 500) is mounted to the first arm 602, and a second robot manipulator 608 (e.g., another robot manipulator 500) is mounted to the second arm 604.

[0075] As described above with reference to FIG. 5. the robot manipulators (606, 608) are designed to match the visual and mechanical properties of the gripper devices (e.g., 100 or 300) described in previous figures. Each manipulator incorporates identical injection-molded fingers with the samevisual tracking features and mask design as the gripper device to maintain a matching visual profile when viewed by the robot's camera system.

[0076] In FIG. 6, camera assemblies 106 can be seen mounted to the manipulators (606. 608). The camera assemblies 106 can be mounted to match their position on the corresponding gripper device, such that the image data from each of the robot's camera assemblies 106 is a visual match to the image data collected from a gripper device used to perform the same manipulation task. In some examples, the field of view of the camera assemblies 106 can be the same on the gripper device and the corresponding robot manipulator; in other examples, the field of view may differ between the two devices, and may be aligned or reconciled by filtering, cropping, or otherwise adjusting the image data from the gripper device (prior to training) and / or the robot manipulator (prior to providing the image data as input to the trained model during inference and robot control).

[0077] The robot 600 demonstrates the hardware-agnostic nature of the manipulator design. The robot manipulators can be mounted to different robot arm platforms while maintaining the same mechanical and visual properties that correspond to the gripper devices. The model can be trained to reproduce the manipulation task in the context of the robot 600 by translating the model output to a control scheme for the robot arms, which generally allows the position of the robot's arms and the actuators for the manipulators to be precisely controlled to match hand position and grip width outputs of the model. This correspondence between the human-operated gripper devices and robot manipulators enables reliable transfer of demonstrated manipulation behaviors to the robot system, regardless of the details of the robot's specific control scheme.

[0078] The dual-arm configuration allows the robot 600 to perform bimanual manipulation tasks that have been demonstrated using a pair of gripper devices. The robot 600 preserves some or all of the mechanical advantages of the gripper device design, including the transmissionless rotary actuation and precise grip width control through low-backlash gearing.

[0079] FIG. 7 illustrates a top view of different designs for fingertips 702 for a gripper device and robot manipulator with various honeycomb structures 710 providing a visually compliant region and with various fingernail structures.

[0080] In each example design, the fingertips 702 incorporate specialized features including a pad 704 and a fingernail structure 706. The pad 704 comprises a visually compliant region 708 with honeycomb structures 710 that deform in response to applied forces.

[0081] The fingertips 702 can be manufactured through a two-shot injection molding process using two different materials - in some examples, a yellow or orange deformable material is used for the visually compliant regions 708, and a stiffer, differently-colored (e.g., black) material is used for the main body 128 and the fingernail structures 706. This manufacturing approach enables the honeycomb structures 710 to be integrally formed into the fingertip design rather than added afterward.

[0082] The visually compliant regions 708 are strategically positioned to be visible in the field of view of the camera assembly, providing visual indication of forces applied during manipulation tasks without requiring force sensors. The honeycomb structures 710 are designed to deform in predictable patterns that can be tracked by the camera system. In some examples, the honeycomb structures 710 also contribute to the deformation patterns of the fingertips 702 to improve gripping or pushing engagement with objects.

[0083] Multiple fingertip designs are shown with varying patterns of honeycomb structures 710, enabling different types of force visualization and gripping capabilities. The designs incorporate both fine manipulation features through the fingernail structure 706 and compliant gripping through the deformable pad 704.

[0084] Thus, in some examples, the fingers include visually compliant regions 708 with honeycomb structures 710 that visibly deform in response to forces applied during gripping operations. The honeycomb structures 710 are designed to deform in predictable patterns that can be tracked by thecamera system. The deformation patterns provide visual indication of applied forces without requiring separate force sensors.

[0085] FIG. 8 illustrates a perspective view of fingertips 702 of a gripper device or robot manipulator with honeycomb structures and fingernail features. The example shown in FIG. 8 corresponds to one of the example designs shown in FIG. 7, from a perspective view showing the inner surface of the pad 704.

[0086] FIG. 9 illustrates a perspective view of a visually compliant region of a fingertip 702 of a gripper device or robot manipulator showing honeycomb structures deforming in response to force from engaging a surface 902, such as a tabletop or other work surface.

[0087] The example shown in FIG. 9 corresponds to the fingertip design shown in FIG. 8. The honeycomb structures 710 visibly change their shape to show horizontal shearing forces and downward forces of the fingertip 702 against the surface 902.

[0088] FIG. 10 illustrates different examples of grip surface textures 1002 for the inner surfaces of the gripper device fingers.

[0089] The grip surface textures 1002 include various patterns such as horizontal ridges, vertical ridges, checkerboard patterns, and arrays of rounded bumps or other protrusions, all arranged at various periodicities and having various dimensions. These different textures are manufactured through the injection molding process to provide varying levels of frictional engagement with objects being manipulated.

[0090] The textured surfaces can be formed from the same deformable material (e.g., a yellow or orange deformable polymer material) used for the visually compliant regions and tracking features of the fingers, contrasting with the differently-colored (e.g.. black) main body. The textures can be integrally formed during the manufacturing process rather than being added afterward, e.g., by a two-shot injection molding process.

[0091] The different patterns of the textures 1002 can enable improved gripping capabilities for different types of manipulation tasks. When combined with the multi -antipodal gripping surfaces and deformable materials of the fingers, these textured surfaces can enhance frictional engagement as the fingers rotate toward each other during gripping operations.

[0092] FIG. 11 illustrates a block diagram of a system 1100 for training a computational model 1114 to operate a robot 600 using data collected from gripper devices (such as gripper device 100 or gripper device 300).

[0093] A user 1106 operates a first gripper device 1102 in a left hand and a second gripper device 1104 in a right hand to generate image data 1108 (as well as. in some cases, additional data such as position data from an IMU. audio data from a microphone, location data, time stamp data, device identifier data, user identifier data, task identifier data, data owner identifier data, and so on). The image data 1108 (and optionally other data) from both devices is collected and provided to a sorting and preprocessing stage 1110. In some examples, the sorting and preprocessing stage 1110 automatically pairs and synchronizes the data streams from the first gripper device 1102 and second gripper device 1104 used for a given manipulation task based on the received data. In some examples, the sorting and preprocessing stage 1110 receives inputs of a large number of data streams from a large number of gripper devices being used to collect training data, and the sorting and preprocessing stage 1110 includes a specialized algorithm to sort and match data from pairs of devices used jointly for a given task. For example, the sorting and preprocessing stage 1110 can be performed on a computing system receiving hundreds or thousands of packages of data from individual devices sent over a network, or read from tangible media such as memory cards, each package representing the data collected by a single gripper device during one or more manipulations tasks.

[0094] In some examples, the sorting and preprocessing stage 1110 can process the image data 1108 from the camera and position data from the IMU of each device to perform SLAM, and the spatial maps generated bySLAM can be used to pair the data from the two devices used on the left and right hands of the user 1106. In some examples, this pairing or matching is performed after filtering the data using rough filtering criteria, such as user identifier data, time stamp data, and / or geographic location data. It will be appreciated that various techniques can be used to sort and match the pairs of data streams from individual devices to identify pairs of devices jointly performing a task in the same place at the same time on the hands of the same user 1106.

[0095] In some examples, the system 1100 automatically pairs corresponding left and right hand video files from a large collection using both visual and audio features extracted from the recordings. The pairing process performed by the sorting and preprocessing stage 1110 can analyze scene content, timing information, and / or synchronized audio cues to identify videos captured simultaneously during the same manipulation task. This automated approach can enable efficient processing of data from large device fleets without requiring manual tracking of device pairs or recording sessions.

[0096] In some examples, the sorting and preprocessing stage 1110 uses algorithms or machine learning models trained or configured to automatically determine whether video data was captured from a left hand or right hand device based on visual characteristics of the manipulation tasks. This automated hand classification operation allows the manufacturing of ambidextrous gripper devices that can be used interchangeably on either hand, as described above, thereby simplifying production and deployment. The right-left classification capability of the sorting and preprocessing stage 1110 can support scaling to large numbers of devices by eliminating the need to manually track and pair specific left and right hand units.

[0097] After two data streams are paired, the sorting and preprocessing stage 1110 performs further preprocessing of the data to prepare it for use in the training process. In some examples, audio data or other data sources can be used to synchronize the paired data streams, e.g., by temporally aligning a pair of corresponding audio cues (such as a loud clapping sound or otheracoustically distinct sound used to initiate a data collection session). In some examples, distracting content can be filtered from the image data 1108, such as moving objects (including humans) shown in the field of view of the camera assembly. In some examples, any portions of the user's hands extending beyond the mask 120 can be visually masked or filtered out of the image data 1108 to more closely match the visual profile of the gripper device to a visual profile of the robot manipulators to be used by the robot.

[0098] In some examples, the sy stem masks out humans or other moving objects from the camera images before performing SLAM processing, because some SLAM algorithms operate most effectively on static environments. The masking process removes dynamic elements that could otherwise interfere with accurate spatial tracking. This approach can improve the robustness and reliability of the SLAM operations for data collected in environments with human activity.

[0099] In some examples, SLAM is performed for the first time or for a second time after the data streams are paired; for example, a joint SLAM operation performed after pairing can include generation of a spatial map that includes the position of both gripper devices within the mapped space. The sorting and preprocessing stage 1110 can perform SLAM by jointly processing data from both left and right hand cameras (and position sensors) to generate a unified spatial map. The use of joint SLAM processing can improve tracking accuracy for each hand by incorporating visual features observed from multiple perspectives into a single consistent representation of the environment. This approach can reduce the need for external fiducial markers by leveraging naturally occurring environmental features like corners of walls, floors, and ceilings that are visible in the expanded field of view of both cameras.

[0100] In addition to SLAM data, the sorting and preprocessing stage 1110 can generate other types of data by processing the video data, such as a stream of grip width data for each gripper device (based on the video data from each camera assembly), a stream of visual deformation parameters relating to the visually compliant regions of each gripper device, etc. Insome examples, the image data 1108 and / or other data is transformed in various ways by the sorting and preprocessing stage 1110, such as cropping, downsampling, filtering, etc.

[0101] In some examples, the sorting and preprocessing stage 1110 performs calibration and tracking preprocessing operations that convert the observed positions of the visual tracking dots into precise radian values representing the grip width. The tracking operations can account for camera perspective and finger geometry to maintain accurate measurements across the full range of motion. These radian values provide a standardized representation of grip states that can be directly mapped to robot manipulator control; in some examples, these radian values can be provided to the model during training in addition to the preprocessed image data 1108.

[0102] After the sorting and preprocessing stage 1110 completes its sorting and preprocessing operations, the resulting preprocessed data is stored as training data 1112 that is used by a model training stage 1116 to train a computational model 1114. The model training stage 1116 trains the computational model 1114 to produce a trained computational model 1118. The training incorporates feature engineering to extract relevant information from the image data 1108 and / or other data streams, and to adjust learnable parameter values of the computational model 1114 to generate a trained computational model 1118 trained to perform the manipulation task shown in the training data 1112 based on robot data input streams formatted to match the training data 1112. An example machine learning pipeline, describing general approaches that can be used to train a computational models such as the computational model 1114, is described below with reference to FIG. 13 and FIG. 14.

[0103] After training is complete, the trained computational model 1118 is deployed to perform inference by operating a robot 600. During inference, the trained computational model 1118 receives robot image data 1120 from a first robot manipulator 606 (such as robot manipulator 500) mounted on a first arm 602 of the robot 600 and a second robot manipulator 608 mounted on a second arm 604 of the robot 600. The trained computational model1118 processes the robot image data 1120 (and optionally other data, such as IMU position data) as inputs to generate output control data 1122 to operate the robot manipulators. The robot manipulators match the mechanical and visual properties of the gripper devices to enable reliable transfer of demonstrated behaviors.

[0104] In some examples, the data received from the robot manipulators is preprocessed in various ways to match its format and content to the format and content of the training data 1112 used to train the model. For example, a field of view of the robot image data 1120 can be cropped, and / or the robot image data 1120 can be filtered or resampled, to match the image data format on which the model was trained. The robot image data 1120 can be processed to generate grip width data and / or visual deformation data for input to the model to match corresponding inputs from the training data 1112. Other preprocessing of the robot sensor data, such as SLAM operations, can be performed to generate other inputs to the trained computational model 1118 for generating the control data 1122.

[0105] In some examples, the control data 1122 encodes a temporal stream of spatial coordinates for movement of the robot manipulators, including a grip width value as well as position data for each robot manipulator (e.g., six-dimensional x-y-z-pitch-roll-yaw information). The individual robot platform used to implement the robot 600 can translate this coordinate and grip width data into control signals for individual effectors or actuators to move the arms (602, 604) and actuators 502 to achieve the desired positions of the robot manipulators (606, 608).

[0106] The system 1100 provides an automated pipeline for collecting demonstration data at scale. The pipeline handles tasks such as pairing corresponding left and right hand data, synchronizing video streams, and masking out humans from the video data to improve spatial tracking robustness. The trained model can then control robot manipulators across different platforms while maintaining consistent manipulation capabilities.

[0107] The various operations and stages of the system 1100 can be implemented using suitable combinations of hardware and software toautomate the collection, sorting, and preprocessing of training data, the training of models to perform specific manipulations tasks, and the deployment and control of robots 600 equipped with trained computational models 1118 for performing the manipulations tasks. In some examples, computer-readable media can include instructions and other data (such as the computational model 1114 or trained computational model 1118) to automate the collection and sorting of training data, the training of computational models using the captured data, and the use of the trained models to control robots.

[0108] In some examples, a non-transitory computer-readable storage medium includes instructions that, when executed, cause a system to control a robot using a trained computational model. The model is trained using data captured by gripper devices during manipulation tasks, enabling the robot to replicate demonstrated behaviors.

[0109] Example methods, described below with reference to FIG. 12, include automated preprocessing of the training data and deployment of trained models across different robot platforms.

[0110] FIG. 12 illustrates a flowchart of operations of a method 1200 for training a computational model to operate a robot using data from gripper devices.

[0111] Although the example method 1200 depicts a particular sequence of operations, the sequence may be altered without departing from the scope of the present disclosure. For example, some of the operations depicted may be performed in parallel or in a different sequence that does not materially affect the function of the method 1200. In other examples, different components of an example device or system that implements the method 1200 may perform functions at substantially the same time or in a specific sequence.

[0112] The method 1200 is described below with reference to the example gripper devices (100 or 300) and robot manipulators 500 described herein as part of the example system 1100. However, it will be appreciated that someexamples of the method 1200 can be performed using devices, systems, and techniques that differ from those described above.

[0113] The method 1200 includes an operation 1202 of mounting first and second gripper devices to a user's hands. The gripper devices (e.g.. 100 or 300) include rotatable fingers actuated by the user's thumb and fingers, and camera assemblies to capture image data of manipulation tasks.

[0114] In operation 1204, the gripper devices are operated by the user to capture image data showing a manipulation task performed by the user. The camera assemblies capture both the manipulation workspace and environmental context while the user performs demonstration tasks. In some examples, additional data can be captured as well, as described above.

[0115] Operation 1206 involves training a computational model using the captured image data. The training process can include feature engineering to extract relevant information from the visual and sensor data streams collected during the demonstrations, which is used to adjust learnable parameter of a computational model 1114 to generate a trained computational model 1118. In some examples, the image data is sorted, preprocessed, and / or combined with additional data to generate the training data 1112 used to train the model, as described above.

[0116] In operation 1208, inference is performed using the trained computational model to operate a robot having a pair of robot manipulators mounted to a pair of arms. The robot manipulators match the mechanical and visual properties of the gripper devices to enable reliable transfer of demonstrated behaviors. The trained model processes robot image data to generate control signals for operating the robot manipulators to perform manipulation tasks, relying on visual feedback (and / or other feedback) generated by the robot 600 and the cameras and other sensors of the robot manipulators 500.

[0117] The method 1200 enables automated collection and processing of demonstration data to train robots to perform manipulation tasks. The end-to-end pipeline handles tasks such as pairing corresponding left and right hand data, synchronizing video streams, masking out distractors from thevideo data to improve spatial tracking robustness, generating supplemental data (e.g,., SLAM, grip width) to assist the training and inference processes, etc.

[0118] FIG. 13 and FIG. 14 below provide examples of training and deployment pipelines for machine learning models, suitable for application to the method 1200 and system 1100 described herein.MACHINE-LEARNING PIPELINE

[0119] FIG. 13 is a flowchart depicting a machine-learning pipeline 1300, according to some examples. The machine-learning pipeline 1300 may be used to generate a trained model, for example the trained machine-learning program 1402 of FIG. 14, to perform operations associated with searches and query responses.Overview

[0120] Broadly, machine learning may involve using computer algorithms to automatically leam patterns and relationships in data, potentially without the need for explicit programming. Machine learning algorithms can be divided into three main categories: supervised learning, unsupervised learning, and reinforcement learning.• Supervised learning involves training a model using labeled data to predict an output for new, unseen inputs. Examples of supervised learning algorithms include linear regression, decision trees, and neural networks.• Unsupervised learning involves training a model on unlabeled data to find hidden patterns and relationships in the data. Examples of unsupervised learning algorithms include clustering, principal component analysis, and generative models like autoencoders.• Reinforcement learning involves training a model to make decisions in a dynamic environment by receiving feedback in the form of rewards or penalties. Examples of reinforcement learning algorithms include Q- learning and policy gradient methods.

[0121] Examples of specific machine learning algorithms that may be deployed, according to some examples, include logistic regression, which is a type of supervised learning algorithm used for binary classification tasks. Logistic regression models the probability of a binary response variable based on one or more predictor variables. Another example ty pe of machine learning algorithm is Naive Bayes, which is another supervised learning algorithm used for classification tasks. Naive Bayes is based on Bayes' theorem and assumes that the predictor variables are independent of each other. Random Forest is another type of supervised learning algorithm used for classification, regression, and other tasks. Random Forest builds a collection of decision trees and combines their outputs to make predictions. Further examples include neural networks, which consist of interconnected layers of nodes (or neurons) that process information and make predictions based on the input data. Matrix factorization is another type of machine learning algorithm used for recommender systems and other tasks. Matrix factorization decomposes a matrix into two or more matrices to uncover hidden patterns or relationships in the data. Support Vector Machines (SVM) are a ty pe of supervised learning algorithm used for classification, regression, and other tasks. SVM finds a hyperplane that separates the different classes in the data. Other types of machine learning algorithms include decision trees, k-nearest neighbors, clustering algorithms, and deep learning algorithms such as convolutional neural networks (CNN), recurrent neural networks (RNN), and transformer models. The choice of algorithm depends on the nature of the data, the complexity of the problem, and the performance requirements of the application.

[0122] The performance of machine learning models is ty pically evaluated on a separate test set of data that was not used during training to ensure that the model can generalize to new, unseen data.

[0123] Although several specific examples of machine learning algorithms are discussed herein, the principles discussed herein can be applied to other machine learning algorithms as well. Deep learning algorithms such as convolutional neural networks, recurrent neural networks, and transformers,as well as more traditional machine learning algorithms like decision trees, random forests, and gradient boosting may be used in various machine learning applications.

[0124] Two example types of problems in machine learning are classification problems and regression problems. Classification problems, also referred to as categorization problems, aim at classifying items into one of several category values (for example, is this object an apple or an orange?). Regression algorithms aim at quantifying some items (for example, by providing a value that is a real number).Training phases

[0125] Generating a trained machine-learning program may include multiple phases that form part of the machine-learning pipeline 1300, including for example the following phases illustrated in FIG. 13:• Data collection and preprocessing 1302: This phase may include acquiring and cleaning data to ensure that it is suitable for use in the machine learning model. This phase may also include removing duplicates, handling missing values, and converting data into a suitable format.• Feature engineering 1304: This phase may include selecting and transforming the training data 1406 to create features that are useful for predicting the target variable. Feature engineering may include (1) receiving features 1408 (e.g., as structured or labeled data in supervised learning) and / or (2) identifying features 1408 (e g., unstructured or unlabeled data for unsupervised learning) in training data 1406.• Model selection and training 1306: This phase may include selecting an appropriate machine learning algorithm and training it on the preprocessed data. This phase may further involve splitting the data into training and testing sets, using cross-validation to evaluate the model, and tuning hyperparameters to improve performance.• Model evaluation 1308: This phase may include evaluating the performance of a trained model (e.g., the trained machine-learning program 1402) on a separate testing dataset. This phase can helpdetermine if the model is overfitting or underfitting and determine whether the model is suitable for deployment.• Prediction 1310: This phase involves using a trained model (e.g., trained machine-learning program 1402) to generate predictions on new, unseen data.• Validation, refinement or retraining 1312: This phase may include updating a model based on feedback generated from the prediction phase, such as new data or user feedback.• Deployment 1314: This phase may include integrating the trained model (e.g., the trained machine-learning program 1402) into a more extensive system or application, such as a web service, mobile app, or loT device. This phase can involve setting up APIs, building a user interface, and ensuring that the model is scalable and can handle large volumes of data.

[0126] FIG. 14 illustrates further details of two example phases, namely a training phase 1404 (e.g., part of the model selection and trainings 1306 of FIG. 13) and a prediction phase 1410 (part of prediction 1310 of FIG. 13). Prior to the training phase 1404, feature engineering 1304 is used to identify features 1408. This may include identifying informative, discriminating, and independent features for effectively operating the trained machine-learning program 1402 in pattern recognition, classification, and regression. In some examples, the training data 1406 includes labeled data, known for preidentified features 1408 and one or more outcomes. Each of the features 1408 may be a variable or attribute, such as an individual measurable property of a process, article, system, or phenomenon represented by a data set (e.g., the training data 1406). Features 1408 may also be of different types, such as numeric features, strings, and graphs, and may include one or more of content 1412, concepts 1414, attributes 1416, historical data 1418 , and / or user data 1420, merely for example.

[0127] In training phase 1404, the machine-learning pipeline 1400 uses the training data 1406 to find correlations among the features 1408 that affect a predicted outcome or prediction / inference data 1422.

[0128] With the training data 1406 and the identified features 1408, the trained machine-learning program 1402 is trained during the training phase 1404 during machine-learning program training 1424. The machine-learning program training 1424 appraises values of the features 1408 as they correlate to the training data 1406. The result of the training is the trained machinelearning program 1402 (e.g., a trained or learned model).

[0129] Further, the training phase 1404 may involve machine learning, in which the training data 1406 is structured (e g., labeled during preprocessing operations). The trained machine-learning program 1402 implements a neural network 1426 capable of performing, for example, classification and clustering operations. In other examples, the training phase 1404 may involve deep learning, in which the training data 1406 is unstructured, and the trained machine-learning program 1402 implements a deep neural network 1426 that can perform both feature extraction andcl assifi cati on / clustering operati ons .

[0130] In some examples, a neural network 1426 may be generated during the training phase 1404. and implemented within the trained machinelearning program 1402. The neural network 1426 includes a hierarchical (e.g., layered) organization of neurons, with each layer consisting of multiple neurons or nodes. Neurons in the input layer receive the input data, while neurons in the output layer produce the final output of the network. Between the input and output layers, there may be one or more hidden layers, each consisting of multiple neurons.

[0131] Each neuron in the neural network 1426 operationally computes a function, such as an activation function, which takes as input the weighted sum of the outputs of the neurons in the previous layer, as well as a bias term. The output of this function is then passed as input to the neurons in the next layer. If the output of the activation function exceeds a certain threshold, an output is communicated from that neuron (e.g., transmitting neuron) to a connected neuron (e.g., receiving neuron) in successive layers. The connections between neurons have associated weights, which define the influence of the input from a transmitting neuron to a receiving neuron.During the training phase, these weights are adjusted by the learning algorithm to optimize the performance of the network. Different types of neural networks may use different activation functions and learning algorithms, affecting their performance on different tasks. The layered organization of neurons and the use of activation functions and weights enable neural networks to model complex relationships between inputs and outputs, and to generalize to new inputs that were not seen during training.

[0132] In some examples, the neural network 1426 may also be one of several different types of neural networks, such as a single-layer feedforward network, a Multilayer Perceptron (MLP), an Artificial Neural Network (ANN), a Recurrent Neural Network (RNN), a Long Short-Term Memory Network (LSTM), a Bidirectional Neural Network, a symmetrically connected neural network, a Deep Belief Network (DBN), a Convolutional Neural Network (CNN), a Generative Adversarial Network (GAN), an Autoencoder Neural Network (AE), a Restricted Boltzmann Machine (RBM), a Hopfield Network, a Self-Organizing Map (SOM), a Radial Basis Function Network (RBFN), a Spiking Neural Network (SNN), a Liquid State Machine (LSM), an Echo State Network (ESN), a Neural Turing Machine (NTM), or a Transformer Network, merely for example.

[0133] In addition to the training phase 1404, a validation phase may be performed on a separate dataset known as the validation dataset. The validation dataset is used to tune the hyperparameters of a model, such as the learning rate and the regularization parameter. The hyperparameters are adjusted to improve the model's performance on the validation dataset.

[0134] Once a model is fully trained and validated, in a testing phase, the model may be tested on a new dataset. The testing dataset is used to evaluate the model's performance and ensure that the model has not overfitted the training data.

[0135] In prediction phase 1410, the trained machine-learning program 1402 uses the features 1408 for analyzing query7data 1428 to generate inferences, outcomes, or predictions, as examples of a prediction / inference data 1422. For example, during prediction phase 1410, the trained machine-learning program 1402 generates an output. Query data 1428 is provided as an input to the trained machine-learning program 1402, and the trained machine-learning program 1402 generates the prediction / inference data 1422 as output, responsive to receipt of the query data 1428.

[0136] In some examples, the trained machine-learning program 1402 may be a generative Al model. Generative Al is a term that may refer to any type of artificial intelligence that can create new content from training data 1406. For example, generative Al can produce text, images, video, audio, code, or synthetic data similar to the original data but not identical.

[0137] Some of the techniques that may be used in generative Al are:• Convolutional Neural Networks (CNNs): CNNs may be used for image recognition and computer vision tasks. CNNs may. for example, be designed to extract features from images by using filters or kernels that scan the input image and highlight important patterns.• Recurrent Neural Networks (RNNs): RNNs may be used for processing sequential data, such as speech, text, and time series data, for example. RNNs employ feedback loops that allow them to capture temporal dependencies and remember past inputs.• Generative adversarial networks (GANs): GNNs may include two neural networks: a generator and a discriminator. The generator network attempts to create realistic content that can “fool” the discriminator network, while the discriminator network attempts to distinguish between real and fake content. The generator and discriminator networks compete with each other and improve over time.• Variational autoencoders (VAEs): VAEs may encode input data into a latent space (e.g., a compressed representation) and then decode it back into output data. The latent space can be manipulated to generate new variations of the output data. VAEs may use self-attention mechanisms to process input data, allowing them to handle long text sequences and capture complex dependencies.• Transformer models: Transformer models may use attention mechanisms to leam the relationships between different parts of inputdata (such as words or pixels) and generate output data based on these relationships. Transformer models can handle sequential data, such as text or speech, as well as non-sequential data, such as images or code.

[0138] In generative Al examples, the output prediction / inference data 1422 include predictions, translations, summaries or media content.

[0139] Described examples of gripper devices and associated systems, methods, and computer-readable media seek to provide technical solutions to one or more example technical problems, such as the following.

[0140] First, existing robotic manipulation systems lack intuitive and natural control mechanisms for collecting demonstration data. The described transmissionless gripper device provides a technical solution by incorporating rotary joints that allow direct actuation by the user's thumb and fingers without intermediate mechanisms. The common rotational mechanism can use gears or other mechanisms to constrain the fingers to rotate jointly, simplifying grip width to a one-dimensional value while maintaining natural manipulation capabilities.

[0141] Second, existing systems struggle to accurately measure applied forces during manipulation without complex and expensive force sensors. The described examples solve this through visually compliant regions with honeycomb structures that visibly deform under applied forces. The deformation patterns are captured by the camera assembly and provide visual indication of forces without requiring separate sensors. The contrasting first-color material of the gripping members (e.g., yellow or orange) against the second-color main body (e.g., black) enables clear tracking of the fingers and their overall deformation patterns, and the honeycomb structures enable clear tracking of deformation of the gripping members close to the inner gripping surfaces.

[0142] Third, existing approaches face challenges in maintaining consistent visual correspondence between training data and robot execution. The described system addresses this through identical injection-molded fingers used in both the gripper devices and robot manipulators, with integrated visual tracking features and masks to occlude the user's fingers. The robotmanipulator precisely matches the mechanical and visual properties of the gripper device through actuator mechanisms such as low-backlash gears, and through the use of identical visual components.

[0143] Fourth, collecting and processing large-scale demonstration data has typically been prohibitively complex. The described system provides an automated pipeline that handles pairing and synchronizing data streams from left and right hands, filtering out unwanted or irrelevant data to minimize distractions, and generating supplemental data streams like SLAM and grip width measurements.

[0144] Fifth, existing systems often require platform-specific implementations. The described examples solve this through a hardwareagnostic design where the robot manipulators can be mounted to different robot platforms while maintaining consistent mechanical and visual properties. The trained computational model generates control data that can be translated into platform-specific actuation commands while preserving the demonstrated behaviors.

[0145] Sixth, existing gripping systems lack ergonomic design features for extended use. The described examples solve this through an angled offset between the human hand axis and the gripper device finger axis for improved ergonomics during tabletop manipulation, positioning of the camera's center of mass over the operator's hand to minimize torque, and inclusion of padded sections on exterior finger interfaces.

[0146] Seventh, other systems struggle with grip surface durability and cost. The described examples address this problem through overmolded fingertips with a wrap-around design replacing expensive, low-life grip tape, and incorporation of various grip texture options integrally formed into the fingertip design through injection molding.

[0147] Eighth, existing systems lack versatile gripping capabilities. The described examples solve this through multi-stiffness fingertips that combine compliant finger pads with hard ‘“fingernail” structures for different manipulation capabilities, and a contoured power grasp regions designed for antipodal grasping of various object sizes.

[0148] Ninth, existing gripper designs present obstacles in confined spaces. The described examples address this through a narrow profile design enabled by rotary joints rather than linear rails, allowing the device to compress to a minimal width for insertion into tight spaces while maintaining full functionality.

[0149] Tenth, traditional manufacturing approaches result in inconsistent device performance. The described examples solve this through fully injection-molded components including camera brackets, base plates, and fingers, with identical injection-molded fingers used on both gripper devices and robot manipulators to ensure perfect correspondence.

[0150] Eleventh, existing systems typically require complex calibration between training devices and robots. The described examples address this through modular design with injection-molded camera brackets and base plates that ensure consistent positioning of cameras and fingers across devices, enabling reliable transfer of demonstrated behaviors.

[0151] Twelfth, existing approaches lack robust visual tracking capabilities. The described examples solve this through strategic placement of tracking fiducial dots to enable precise measurement of finger positions and gripper width, with high-contrast markers integrally formed into the mold shape rather than added afterward.

[0152] Thirteenth, existing systems typically require separate force sensing hardware. The described examples address this through strain-observable materials that visibly deform under pressure, providing visual force feedback through camera-observable deformation patterns in the honeycomb structures.

[0153] These technical solutions can be configured to work together to enable efficient collection of high-quality demonstration data and reliable transfer of demonstrated behaviors to robotic systems. The mechanical design, visual tracking capabilities, automated data processing pipeline, and hardware-agnostic approach combine to address fundamental challenges in robot learning from human demonstration.

[0154] Other technical problems may be addressed by various examples described herein, as will be appreciated by a skilled person.

[0155] In view of the above-described implementations of subject matter this application discloses the following list of examples, wherein one feature of an example in isolation or more than one feature of an example, taken in combination and, optionally, in combination with one or more features of one or more further examples are further examples also falling within the disclosure of this application.

[0156] Example 1 is a system for generating training data for training a computational model to control a robot, the system comprising: a gripper device mountable to a user's hand, the gripper device comprising: a first finger rotatable inward by inward force applied by a thumb of the hand; a second finger rotatable inward by inward force applied by one or more fingers of the hand; and a camera assembly mounted to the gripper device to capture image data over a field of view comprising at least a portion of the first finger, at least a portion of the second finger, and at least a portion of a workspace extending forward from the first finger and second finger.

[0157] In Example 2, the subject matter of Example 1 includes, wherein the gripper device further comprises: a common rotational mechanism constraining the first finger and second finger to rotate jointly such that radial positions of the first finger and second finger relative to the gripper device are both defined by a one-dimensional grip index.

[0158] In Example 3. the subject matter of Example 2 includes, wherein: the common rotational mechanism comprises two rotational joints coupled to gears to constrain the fingers to rotate jointly.

[0159] In Example 4. the subject matter of Examples 2-3 includes, wherein the gripper device further comprises: visual tracking features integrally formed with the first finger and second finger to present fiducial reference points in the field of view of the camera assembly indicative of the grip index.

[0160] In Example 5, the subject matter of Examples 1-4 includes, wherein: the first finger and second finger mirror each other's structure topermit a user to use the gripper device on either hand, the first finger being a right finger when worn on a left hand, the first finger being a left finger when worn on a right hand.

[0161] In Example 6, the subject matter of Example 5 includes, wherein: the structure of the first finger and the structure of the second finger are formed by injection molding.

[0162] In Example 7, the subject matter of Examples 1-6 includes, wherein: the field of view of the camera assembly further comprises at least a portion of an environment outside of the workspace, comprising line of sight to at least one feature of the environment at more than a 90 degree displacement from an axis extending forward from the gripper device.

[0163] In Example 8, the subject matter of Example 7 includes, wherein: the camera assembly comprises a 360 degree camera.

[0164] In Example 9. the subject matter of Examples 1-8 includes, wherein: a center of mass of the camera assembly is positioned above the hand.

[0165] In Example 10, the subject matter of Examples 1-9 includes, wherein: the first finger and second finger each comprises one or more visually compliant regions visible in the field of view of the camera assembly and configured to visibly deform in response to forces applied to inner surfaces of the fingers.

[0166] In Example 11, the subject matter of Example 10 includes, wherein: the one or more visually compliant regions comprise honeycomb structures.

[0167] In Example 12, the subject matter of Examples 1-11 includes, wherein the gripper device further comprises: a mask to occlude the one or more fingers and the thumb in the field of view of the camera assembly.

[0168] In Example 13, the subject matter of Examples 1-12 includes, wherein the gripper device further comprises: a biasing mechanism to bias the first finger and second finger away from each other; and one or more straps to secure the gripper device to the user's hand.

[0169] In Example 14, the subject matter of Example 13 includes, wherein: the one or more straps comprises: a strap to secure the first finger to the thumb; a strap to secure the second finger to at least one of the one or more fingers; and a strap to secure the gripper device to a wrist or palm of the hand.

[0170] In Example 15, the subject matter of Examples 1-14 includes, wherein the gripper device further comprises: one or more position sensors to generate position data representative of a position of the gripper device.

[0171] In Example 16, the subject matter of Examples 1-15 includes, wherein: the first finger and second finger each comprises: a deformable gripping member defining an inner surface, the gripping member comprising a plurality of zones having different levels of stiffness; the plurality of zones comprises a fingernail structure at a front end of the gripping member having a higher level of stiffness than others of the plurality of zones; the inner surfaces of the gripping members of the two fingers define a multi-antipodal pair of gripping surfaces wherein different pairs of corresponding points on the two inner surfaces are parallel to each other at different radial positions of the two fingers; and at least a portion of the inner surface is textured to improve friction engagement with an object gripped by the gripper device.

[0172] In Example 17, the subject matter of Examples 1-16 includes, a robot manipulator mountable to an arm of the robot, the robot manipulator comprising: a pair of robot manipulator fingers rotatably mounted to the robot manipulator to rotate toward each other to grip objects; a robot manipulator camera assembly to capture image data over a robot field of view; an actuator comprising low-backlash metal gears, the actuator constraining rotation of the pair of robot manipulator fingers to joint rotation such that radial positions of the pair of robot manipulator fingers are both defined by a one-dimensional grip index; and visual tracking features integrally formed with the pair of robot manipulator fingers to present fiducial reference points in the field of view of the robot manipulator camera assembly indicative of the grip index; the pair of robot manipulator fingers having a visual profile in the robot field of view, at a plurality of radialpositions in rotation toward each other, that matches a visual profile of the first finger and second finger in the field of view of the camera assembly of the gripper device at corresponding radial positions; the robot manipulator being configured for mounting to multiple robot platforms.

[0173] Example 18 is a method comprising: mounting a gripper device to a user's hand, the gripper device comprising: a first finger rotatable inward by inward force applied by a thumb of the hand; a second finger rotatable inw ard by inward force applied by one or more fingers of the hand; and a camera assembly mounted to the gripper device to capture image data over a field of view comprising at least a portion of the first finger, at least a portion of the second finger, and at least a portion of a workspace extending forward from the first finger and second finger; operating the gripper device to capture the image data showing a manipulation task performed by the user; training a computational model using the captured image data; performing inference using the trained computational model to operate a robot having a robot manipulator mounted to an arm of the robot, the robot manipulator comprising: a pair of robot manipulator fingers rotatably mounted to the robot manipulator to rotate toward each other to grip objects; and a robot manipulator camera assembly to capture image data over a robot field of view; the pair of robot manipulator fingers having a visual profile in the robot field of view, at a plurality of radial positions in rotation toward each other, that matches a visual profile of the first finger and second finger in the field of view of the camera assembly of the gripper device at corresponding radial positions.

[0174] In Example 19, the subject matter of Example 18 includes, mounting a second gripper device to the user's other hand; and operating the second gripper device during the manipulation task to capture second hand image data; wherein: the training of the computational model uses the captured image data and the captured second hand image data; and the robot has a second robot manipulator mountable to a second arm of the robot.

[0175] Example 20 is a non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that whenexecuted by at least one processor of a system, cause the system to perform operations comprising: controlling a robot using a trained computational model; the computational model being trained using a training dataset comprising image data showing a manipulation task performed by a user's hand with a gripper device comprising: a first finger rotatable inward by inward force applied by a thumb of the hand; a second finger rotatable inward by inward force applied by one or more fingers of the hand; and a camera assembly mounted to the gripper device to capture the image data over a field of view comprising at least a portion of the first finger, at least a portion of the second finger, and at least a portion of a workspace extending forward from the first finger and second finger.

[0176] Example 21 is at least one machine-readable medium including instructions that, when executed by processing circuitry, cause the processing circuitry to perform operations to implement of any of Examples 1-20.

[0177] Example 22 is an apparatus comprising means to implement of any of Examples 1-20.

[0178] Example 23 is a system to implement of any of Examples 1-20.

[0179] Example 24 is a method to implement of any of Examples 1-20.

[0180] Other technical features may be readily apparent to one skilled in the art from the figures, descriptions, and claims herein.

[0181] TERM EXAMPLES

[0182] “Component” refers, for example, to a device, physical entity, or logic having boundaries defined by function or subroutine calls, branch points, APIs, or other technologies that provide for the partitioning or modularization of particular processing or control functions. Components may be combined via their interfaces with other components to carry out a machine process. A component may be a packaged functional hardware unit designed for use with other components and a part of a program that usually performs a particular function of related functions. Components may constitute either software components (e g., code embodied on a machine-readable medium) or hardware components. A "hardware component" is atangible unit capable of performing certain operations and may be configured or arranged in a certain physical manner. In various examples, one or more computer systems (e.g., a standalone computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or a group of processors) may be configured by software (e.g., an application or application portion) as a hardware component that operates to perform certain operations as described herein. A hardware component may also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component may include dedicated circuitry or logic that is permanently configured to perform certain operations. A hardware component may be a special-purpose processor, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). A hardware component may also include programmable logic or circuitry that is temporarily configured by software to perform certain operations. For example, a hardware component may include software executed by a general-purpose processor or other programmable processors. Once configured by such software, hardware components become specific machines (or specific components of a machine) uniquely tailored to perform the configured functions and are no longer general-purpose processors. It will be appreciated that the decision to implement a hardware component mechanically, in dedicated and permanently configured circuitry, or in temporarily configured circuitry (e.g., configured by software), may be driven by cost and time considerations. Accordingly, the phrase "hardware component"(or "hardware-implemented component") should be understood to encompass a tangible entity, be that an entity that is physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain manner or to perform certain operations described herein. Considering examples in which hardware components are temporarily configured (e.g., programmed), each of the hardware components need not be configured or instantiated at any one instance in time. For example, where a hardware component comprises a general-purpose processor configured by software to become a special-purpose processor, the general-purpose processor may be configured as respectively different special-purpose processors (e.g., comprising different hardware components) at different times. Software accordingly configures a particular processor or processors, for example, to constitute a particular hardware component at one instance of time and to constitute a different hardware component at a different instance of time. Hardware components can provide information to, and receive information from, other hardware components. Accordingly, the described hardware components may be regarded as being communicatively coupled. Where multiple hardware components exist contemporaneously, communications may be achieved through signal transmission (e.g.. over appropriate circuits and buses) between or among two or more of the hardware components. In examples in which multiple hardware components are configured or instantiated at different times, communications between such hardware components may be achieved, for example, through the storage and retrieval of information in memory structures to which the multiple hardware components have access. For example, one hardw are component may perform an operation and store the output of that operation in a memory device to w hich it is communicatively coupled. A further hardw are component may then, at a later time, access the memory device to retrieve and process the stored output. Hardware components may also initiate communications with input or output devices, and can operate on a resource (e.g., a collection of information). The various operations of example methods described herein may be performed, at least partially, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors may constitute processor-implemented components that operate to perform one or more operations or functions described herein. As used herein, "processor-implemented component7’ refers to a hardware component implemented using one or more processors. Similarly, the methods described herein may be at least partially processor-implemented, with a particular processor or processors being an example of hardware. For example, at least some of the operations of a method may beperformed by one or more processors or processor-implemented components. Moreover, the one or more processors may also operate to support performance of the relevant operations in a "cloud computing" environment or as a "software as a service" (SaaS). For example, at least some of the operations may be performed by a group of computers (as examples of machines including processors), with these operations being accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., an API). The performance of certain of the operations may be distributed among the processors, not only residing within a single machine, but deployed across a number of machines. In some examples, the processors or processor-implemented components may be located in a single geographic location (e.g., within a home environment, an office environment, or a server farm). In other examples, the processors or processor-implemented components may be distributed across a number of geographic locations.

[0183] “Computer-readable storage medium” refers, for example, to both machine-storage media and transmission media. Thus, the terms include both storage devices / media and carrier waves / modulated data signals. The terms “machine-readable medium,” “computer-readable medium” and “device-readable medium” mean the same thing and may be used interchangeably in this disclosure.

[0184] “ Machine storage medium” refers, for example, to a single or multiple storage devices and media (e.g., a centralized or distributed database, and associated caches and servers) that store executable instructions, routines and data. The term shall accordingly be taken to include, but not be limited to. solid-state memories, and optical and magnetic media, including memory internal or external to processors. Specific examples of machine-storage media, computer-storage media and devicestorage media include non-volatile memory', including by way of example semiconductor memory devices, e.g.. erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory' (EEPROM), FPGA, and flash memory devices; magnetic disks such asinternal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks The terms “machine-storage medium,’" “devicestorage medium,” “computer-storage medium” mean the same thing and may be used interchangeably in this disclosure. The terms “machine-storage media,” “computer-storage media,” and “device-storage media” specifically exclude carrier waves, modulated data signals, and other such media, at least some of which are covered under the term “signal medium.”

[0185] “N on-transitory computer-readable storage medium” refers, for example, to a tangible medium that is capable of storing, encoding, or carrying the instructions for execution by a machine.

[0186] “Processor” refers to any circuit or virtual circuit (a physical circuit emulated by logic executing on an actual processor) that manipulates data values according to control signals (e.g., “commands”, “op codes”, “machine code”, etc.) and which produces corresponding output signals that are applied to operate a machine. A processor may, for example, be a Central Processing Unit (CPU), a Reduced Instruction Set Computing (RISC) processor, a Complex Instruction Set Computing (CISC) processor, a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Radio-Frequency Integrated Circuit (RFIC) or any combination thereof. A processor may further be a multi-core processor having two or more independent processors (sometimes referred to as “cores”) that may execute instructions contemporaneously.

[0187] “Signal medium” refers, for example, to any intangible medium that is capable of storing, encoding, or carrying the instructions for execution by a machine and includes digital or analog communications signals or other intangible media to facilitate communication of software or data. The term “signal medium” shall be taken to include any form of a modulated data signal, carrier wave, and so forth. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a matter as to encode information in the signal. The terms‘‘transmission medium” and “signal medium” mean the same thing and may be used interchangeably in this disclosure.

[0188] As used in this disclosure, phrases of the form “at least one of an A, a B, or a C,” “at least one of A, B, or C,” “at least one of A, B, and C,” and the like, should be interpreted to select at least one from the group that comprises “A, B, and C.” Unless explicitly stated otherwise in connection with a particular instance in this disclosure, this manner of phrasing does not mean “at least one of A, at least one of B, and at least one of C.” As used in this disclosure, the example “at least one of an A, a B, or a C,” would cover any of the following selections: {A], {B}, {C}, {A, B}, {A, C}, {B, C}, and {A, B, C}.

[0189] Unless the context clearly requires otherwise, throughout the description and the claims, the words “comprise,” “comprising,” and the like are to be construed in an inclusive sense, as opposed to an exclusive or exhaustive sense, e.g., in the sense of “including, but not limited to.”

[0190] As used herein, the terms “connected,” “coupled,” or any variant thereof means any connection or coupling, either direct or indirect, between two or more elements; the coupling or connection between the elements can be physical, logical, or a combination thereof.

[0191] Additionally, the words “herein,” “above,” “below,” and words of similar import, when used in this application, refer to this application as a whole and not to any portions of this application. Where the context permits, words using the singular or plural number may also include the plural or singular number respectively.

[0192] The word “or” in reference to a list of two or more items, covers all the following interpretations of the word: any one of the items in the list, all the items in the list, and any combination of the items in the list. Likewise, the term “and / or” in reference to a list of two or more items, covers all the following interpretations of the word: any one of the items in the list, all the items in the list, and any combination of the items in the list.

[0193] The various features, operations, or processes described herein may be used independently of one another, or may be combined in various ways.All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. In addition, certain method or process blocks may be omitted in some implementations.

[0194] Although some examples, e.g., those depicted in the drawings, include a particular sequence of operations, the sequence may be altered without departing from the scope of the present disclosure. For example, some of the operations depicted may be performed in parallel or in a different sequence that does not materially affect the functions as described in the examples. In other examples, different components of an example device or system that implements an example method may perform functions at substantially the same time or in a specific sequence.

Claims

CLAIMSWhat is claimed is:

1. A system for generating training data for training a computational model to control a robot, the system comprising:a gripper device mountable to a user's hand, the gripper device comprising:a first finger rotatable inward by inward force applied by a thumb of the hand;a second finger rotatable inward by inward force applied by one or more fingers of the hand; anda camera assembly mounted to the gripper device to capture image data over a field of view comprising at least a portion of the first finger, at least a portion of the second finger, and at least a portion of a workspace extending forward from the first finger and second finger.

2. The system of claim 1, wherein the gripper device further comprises: a common rotational mechanism constraining the first finger and second finger to rotate j ointly such that radial positions of the first finger and second finger relative to the gripper device are both defined by a onedimensional grip index.

3. The system of claim 2, wherein:the common rotational mechanism comprises two rotational joints coupled to gears to constrain the fingers to rotate jointly.

4. The system of claim 2, wherein the gripper device further comprises: visual tracking features integrally formed with the first finger and second finger to present fiducial reference points in the field of view of the camera assembly indicative of the grip index.

5. The system of claim 1, wherein:the first finger and second finger mirror each other's structure to permit a user to use the gripper device on either hand, the first finger being a right finger when worn on a left hand, the first finger being a left finger when worn on a right hand.

6. The system of claim 5, wherein:the structure of the first finger and the structure of the second finger are formed by injection molding.

7. The system of claim 1, wherein:the field of view of the camera assembly further comprises at least a portion of an environment outside of the workspace, comprising line of sight to at least one feature of the environment at more than a 90 degree displacement from an axis extending forward from the gripper device.

8. The system of claim 7, wherein:the camera assembly comprises a 360 degree camera.

9. The system of claim 1, wherein:a center of mass of the camera assembly is positioned above the hand.

10. The system of claim 1, wherein:the first finger and second finger each comprises one or more visually compliant regions visible in the field of view of the camera assembly and configured to visibly deform in response to forces applied to inner surfaces of the fingers.

11. The system of claim 10, wherein:the one or more visually compliant regions comprise honeycomb structures.

12. The system of claim 1, wherein the gripper device further comprises: a mask to occlude the one or more fingers and the thumb in the field of view of the camera assembly.

13. The system of claim 1, wherein the gripper device further comprises: a biasing mechanism to bias the first finger and second finger away from each other; andone or more straps to secure the gripper device to the user's hand.

14. The system of claim 13, wherein:the one or more straps comprises:a strap to secure the first finger to the thumb;a strap to secure the second finger to at least one of the one or more fingers; anda strap to secure the gripper device to a wrist or palm of the hand.

15. The system of claim 1, wherein the gripper device further comprises: one or more position sensors to generate position data representative of a position of the gripper device.

16. The system of claim 1, wherein:the first finger and second finger each comprises:a deformable gripping member defining an inner surface, the gripping member comprising a plurality of zones having different levels of stiffness;the plurality of zones comprises a fingernail structure at a front end of the gripping member having a higher level of stiffness than others of the plurality of zones;the inner surfaces of the gripping members of the two fingers define a multi-antipodal pair of gripping surfaces wherein different pairs of corresponding points on the two inner surfaces are parallel to each other at different radial positions of the two fingers; and at least a portion of the inner surface is textured to improve friction engagement with an object gripped by the gripper device.

17. The system of claim 1, further comprising:a robot manipulator mountable to an arm of the robot, the robot manipulator comprising:a pair of robot manipulator fingers rotatably mounted to the robot manipulator to rotate toward each other to grip objects;a robot manipulator camera assembly to capture image data over a robot field of view;an actuator comprising low-backlash metal gears, the actuator constraining rotation of the pair of robot manipulator fingers to joint rotation such that radial positions of the pair of robot manipulator fingers are both defined by a one-dimensional grip index; and visual tracking features integrally formed with the pair of robot manipulator fingers to present fiducial reference points in the field of view of the robot manipulator camera assembly indicative of the grip index;the pair of robot manipulator fingers having a visual profile in the robot field of view, at a plurality of radial positions in rotation toward each other, that matches a visual profile of the first finger and second finger in the field of view of the camera assembly of the gripper device at corresponding radial positions;the robot manipulator being configured for mounting to multiple robot platforms.

18. A method comprising:mounting a gripper device to a user's hand, the gripper device comprising:a first finger rotatable inward by inward force applied by a thumb of the hand;a second finger rotatable inward by inward force applied by one or more fingers of the hand; anda camera assembly mounted to the gripper device to capture image data over a field of view comprising at least a portion of the first finger, at least a portion of the second finger, and at least aportion of a workspace extending forward from the first finger and second finger;operating the gripper device to capture the image data showing a manipulation task performed by the user;training a computational model using the captured image data; performing inference using the trained computational model to operate a robot having a robot manipulator mounted to an arm of the robot, the robot manipulator comprising:a pair of robot manipulator fingers rotatably mounted to the robot manipulator to rotate toward each other to grip objects; and a robot manipulator camera assembly to capture image data over a robot field of view;the pair of robot manipulator fingers having a visual profile in the robot field of view, at a plurality of radial positions in rotation toward each other, that matches a visual profile of the first finger and second finger in the field of view of the camera assembly of the gripper device at corresponding radial positions.

19. The method of claim 18, further comprising:mounting a second gripper device to the user's other hand; and operating the second gripper device during the manipulation task to capture second hand image data;wherein:the training of the computational model uses the captured image data and the captured second hand image data; andthe robot has a second robot manipulator mountable to a second arm of the robot.

20. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by at least one processor of a system, cause the system to perform operations comprising:controlling a robot using a trained computational model;the computational model being trained using a training dataset comprising image data showing a manipulation task performed by a user's hand with a gripper device comprising:a first finger rotatable inward by inward force applied by a thumb of the hand;a second finger rotatable inward by inward force applied by one or more fingers of the hand; anda camera assembly mounted to the gripper device to capture the image data over a field of view comprising at least a portion of the first finger, at least a portion of the second finger, and at least a portion of a workspace extending forward from the first finger and second finger.