Device for multimodal tactile sensing

Abstract

A device comprising a housing, a deformable element, a camera, and a piezoresistive layer. The housing having a side wall extending between a base and a cover, the base and the cover being spaced apart from one another along a primary axis defined by the housing. The deformable element being disposed at the cover. The camera being disposed at the base, the camera being oriented with its optical axis coincidental with the primary axis, deformation of the deformable element being visually detectable by the camera. The piezoresistive layer including a plurality of piezoresistive elements disposed in an array. The piezoresistive layer is disposed at the inner surface of the deformable element between the deformable element and the camera.

Description

DEVICE FOR MULTIMODAL TACTILE SENSINGRELATED APPLICATION

[0001] This patent application claims benefit of priority to the Singapore patent application no. 10202500529Y, filed on February 28, 2025, the contents of which are hereby incorporated by reference in entirety for all purposes.TECHNICAL FIELD

[0002] The present disclosure relates to sensors and more particularly sensors suitable for tactile sensing.BACKGROUND

[0003] In many robotic manipulation scenarios, visual information alone is insufficient, and force feedback provided by sensors on the end-effector often fails to fully describe the state of the manipulated object or its interaction with the environment. To achieve dexterous manipulation comparable to human capabilities, tactile perception and the processing of tactile signals have become critical areas of research.

[0004] Tactile sensing involves the characterization of physical object properties through multi-contact interactions. While essential for robotic manipulation, medical devices, and human-computer interaction, current sensor architectures are largely bifurcated into three categories: vision-based, force-sensing, and multimodal sensors.

[0005] Conventional tactile sensors struggle to deliver both high-resolution tactile information and high-frequency force feedback, both of which are essential for realtime manipulation tasks, especially dynamic tasks. Vision-based tactile sensors often lack the rapid response rates needed for dynamic interactions, while forcesensing tactile sensors may miss crucial contact surface details.SUMMARY

[0006] A device includes a housing, a deformable element, a camera, and a piezoresistive layer. The housing has a side wall extending between a base and acover. The base and the cover are spaced apart from one another along a primary axis defined by the housing. The deformable element is disposed at the cover. The camera is disposed at the base. The camera is oriented with its optical axis coincidental with the primary axis. Deformation of the deformable element is visually detectable by the camera. The piezoresistive layer includes a plurality of piezoresistive elements disposed in an array, in which the piezoresistive layer is disposed at the inner surface of the deformable element between the deformable element and the camera.

[0007] The device may further include: a processor configured to execute instructions stored in a machine-readable memory to perform a method, in which the method includes: capturing a plurality of images of the deformable element, at a same time instant, each of the plurality of images including a plurality of blind areas, each blind area corresponding to an area of the deformable element that is blocked from the camera view by one or more of the plurality of piezoresistive elements; and combining the plurality of images to obtain a combined image with no blind areas.

[0008] In another aspect, a device for multimodal tactile sensing includes: a housing, a vision-based sensing unit, and a force-based sensing unit. The housing has a side wall extending between a base and a cover, the base and the cover being spaced apart from one another along a primary axis defined by the housing. The vision-based sensing unit includes: a deformable element, the deformable element being disposed at the cover, the deformable element being elastically deformable and transparent; a camera, the camera being disposed at the base, the camera being oriented with its optical axis coincidental with the primary axis, deformation of the deformable element being visually detectable by the camera. The force-based sensing unit includes: a piezoresistive layer, the piezoresistive layer including a plurality of piezoresistive elements disposed in an array, wherein the piezoresistive layer is disposed at the inner surface of the deformable element between the deformable element and the camera. The device may further include a support, the support being a transparent component coupled to the side wall, the piezoresistive layer being supported by the support. The device may further include one or more reflective elements, the one or more reflective elements being disposedat a perimeter of the piezoresistive layer, the camera being configured to capture parts of an image of an inner surface of the deformable element occluded by the plurality of piezoresistive elements. The force-based sensing unit and the visionbased sensing unit may be configured to be simultaneously operable in response to a same force or pressure at an outer surface of the deformable element.BRIEF DESCRIPTION OF THE DRAWINGS

[0001] Various embodiments of the present disclosure will be described with reference to the appended figures.

[0002] FIG. 1A and FIG. 1B show an exploded perspective view and a perspective view respectively of a device for multimodal tactile sensing, according to embodiments of the present disclosure.

[0003] FIG. 2 is a cross-sectional view of the device for multimodal tactile sensing, according to embodiments of the present disclosure.

[0004] FIG. 3 illustrates a simulation showing the blind areas corresponding to three camera views, including a left view, a right view, and a main view.

[0005] FIG. 4 illustrates a comparison of image quality captured by the camera before (top) and after (bottom) improvement.

[0006] FIG. 5A and FIG. 5B show an exploded perspective view and a perspective view respectively of a device for multimodal tactile sensing, according to another embodiment of the present disclosure.

[0007] FIG. 6A and FIG. 6B show a perspective view and an exploded perspective view respectively of a device for multimodal tactile sensing, according to another embodiment of the present disclosure.

[0008] FIG. 7 illustrates a schematic showing the multimodal integration according to sensory mechanism of the present disclosure as a mimic of the human skin.

[0009] FIG. 8 illustrates an integration concept of the device with exemplary parameters, including various configurable parameters, according to various embodiments of the present disclosure.

[0010] FIG. 9 illustrates an experimental setup for force sensing calibration of various prototypes of the present disclosure.

[0011] FIG. 10 illustrates an experimental setup for high-frequency vibration response testing of various prototypes of the present disclosure.

[0012] FIG. 11 illustrates a workflow for the in-sensor occlusion removal test

[0013] FIG. 12 shows a plot of voltage (in voltage or V) as a function of force (Newton or N) illustrating force calibration results with hysteresis error data and a total fitted curve.

[0014] FIG. 13A shows a plot of motor position (in millimeters or mm) as a function of time (in milliseconds or ms) illustrating the movement of a test object subjected to vibration at 25 Hz.

[0015] FIG. 13B shows a plot of motor position (in millimeters or mm) as a function of time (in milliseconds or ms) illustrating the movement of a test object subjected to vibration at 50 Hz.

[0016] FIG. 13C shows a plot of force (in Newton or N) as a function of time (in milliseconds or ms) illustrating the sensor response and ground truth force experienced by a test object subjected to vibration at 25 Hz.

[0017] FIG. 13D shows a plot of force (in Newton or N) as a function of time (in milliseconds or ms) illustrating the sensor response and ground truth force experienced by a test object subjected to vibration at 50 Hz.

[0018] FIG. 14A shows a post-processed image wherein the occlusion caused by the piezoresistive layer has been removed.

[0019] FIG. 14B shows an image presenting the ground truth texture wherein the piezoresistive layer has been physically removed from the experimental setup.

[0020] FIG. 15 illustrates the image processing methodology, wherein marker deviations are evaluated.

[0021] FIG. 16 shows a plot of root mean square error (RMSE) of the marker coordinates (in pixels) as a function of press depth (in millimeters or mm), illustrating the relationship between the x and y coordinate deviation sand the gel compression depth.

[0022] FIG. 17 illustrates an application of the device in the manipulation of deformable objects, according to various embodiments of the present disclosure.DETAILED DESCRIPTION

[0023] The following detailed description is made with reference to the accompanying drawings, showing details and embodiments of the present disclosure for the purposes of illustration and to aid understanding, and not to be limiting. Features that are described in the context of an embodiment may correspondingly be applicable to the same or similar features in the other embodiments, even if not explicitly described in these other embodiments. Additions and / or combinations and / or alternatives as described for a feature in the context of an embodiment may correspondingly be applicable to the same or similar feature in the other embodiments.

[0024] As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0025] The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0026] As used herein, the singular ‘a’ and ‘an’ may be construed as including the plural “one or more” unless apparent from the context to be otherwise.

[0027] The terms "about" and "approximately" as applied to a stated numeric value encompasses the exact value and a reasonable variance as will be understood by one of ordinary skill in the art, and the terms “generally” and “substantially” are to be understood in a comparable manner, unless otherwise specified.

[0028] Some processes may be described in terms of steps merely to aid understanding and / or for convenient reference. The delineation between one step and another step may be described as such merely for convenient reference in the present disclosure. It will be understood that in actual implementation there may not be a clear division or transition from one step to another subsequent step. There may be a certain amount of overlap among the steps and / or more than one step may occur or be performed concurrently in time, etc.

[0029] As used herein, the terms "simultaneous" and “concurrent”, or the like, refer to two or more occurrences (or events) that at least partially overlap in time. The occurrences may or may not start at the same time instant and / or end at the same time instant.

[0030] Terms such as “first”, “second”, etc., are used in the description and claims only for the sake of brevity and clarity, and do not necessarily imply a priority or order, unless required by the context.

[0031] FIG. 1 A is an exploded perspective view of a device 100, according to embodiments of the present disclosure. FIG. 1 B is a perspective view and FIG. 2 is a cross-sectional view of the device 100 of FIG. 1A as assembled.

[0032] The device 100 may be described as multimodal tactile sensor including a vision-based sensing module integrated with a force-based sensing module. Components of the device 100 may contribute toward one or both of vision-based sensing and force-based sensing, simultaneously and / or in different aspects, as will be understood from the following description. The device 100 is capable of multimodal sensing, e.g., capable of various different types of measurements, including but not limited to capturing tactile information. The device 100 may be interchangeably referred to as a multimodal sensor or a multimodal tactile sensing device. In some instances, embodiments of the device 100 may be referred to as "MultiTac".

[0033] The device 100 includes a housing 110. The housing 110 may include a base 120 and a cover 130 spaced apart along a primary axis 101. The primary axis 101 may be defined by the housing 110. The primary axis 101 may be defined as an axis extending through a geometric center of the housing 110. In fabricating a prototype of the device 100 for experimental validation, the housing 110 was made using additive manufacturing or 3D printing. It will be understood that other manufacturing methods may be used to fabricate the housing 110.

[0034] The device 100 includes a camera 200 disposed at the base 120 of the housing 110. The camera 200 may include one USB (Universal Serial Bus) camera 201.

[0035] The device includes a deformable element 600 made of an elastomeric material (hence interchangeably referred to herein as an elastomer layer). The deformable element 600 includes an inner surface 610 and an outer surface 620. The inner surface 610 of the deformable element 600 faces the base 120 of the housing. A pigment layer 700 is disposed on the outer surface 620 of the deformable element 600. The pigment layer 700 is compliant with the deformableelement 600 such that the pigment layer 700 and the deformable element 600 corresponding deform upon contact with an external object. Deformation of the pigment layer 700 can be observed by the camera 200 through the transparent deformable element 600 as a result of differences or a change in the way light is reflected by the pigment layer 700 when deformed and when not deformed.

[0036] The device 100 includes a lighting system 500 which may include a light source such as but not limited to light emitting diodes (LED) 510. A plurality of light emitting diodes 510 may be disposed at the sides of cover 130. A light-guiding lens or light guide 520 may be configured to direct light from the light emitting diodes 510 to provide a more uniform lighting of the deformable element 600. The lighting system 500 is configured to light up the inner surface 610 of the deformable element. Light reflected by elastomer layer 600 / pigment layer 700 may be focused by a lens 530 disposed between the inner surface 610 of the deformable element 600 and the camera 200. The lens 530 may be an acrylic lens. For the purpose of the present disclosure, a "lens" refers to an optical device with photonic properties or functions. The device 100 further includes a mirror system 400 disposed at the side wall 140 of the housing 110. The mirror system 400 may include one or more reflective elements disposed at the side wall 140 of the housing 110. FIG. 1A and FIG. 2 illustrate an example where the mirror system 400 includes two flat reflective elements 410 that are disposed on opposing inclined panels of the side wall 140 of the housing 110, with the reflective elements 410 (reflective surfaces) in the interior of the housing.

[0037] The device 100 includes a piezoelectric module that is disposed to be directly physically contactable by the inner surface 610 of the deformable element 600. The piezoelectric module may include a plurality of piezoresistive elements 320 disposed on one or more piezo-support 310. In some examples, the piezoelectric module includes a plurality of piezoresistive elements 320 aligned in a row on a strip-shaped piezo-support 310 with the plurality of piezoresistive elements 320 spaced equidistance apart from one another. The device 100 may include a plurality of rows of piezoresistive elements 320 disposed on respective ones of a plurality of strip-shaped piezo-support 310. Collectively, the device 100 includes a plurality of piezoresistive elements 320 disposed in a regular orgeometrically-patterned array, all the piezoresistive elements 320 being disposed in a same array plane normal to the primary axis 101. The piezo-support 310 may be made of a transparent polymer or ultra-thin glass. For the sake of brevity, the piezoresistive module may be referred to as a piezoresistive layer 300.

[0038] In some embodiments, the camera 200 may be selected with a camera focal length suitable for focusing on the deformable element 600. In some embodiments, the camera 200 may be selected with a camera focal length such that a separate lens 530 is not required to provide the necessary focus.

[0039] In some embodiments, the lens 530 double up as a support 800 for the piezoelectric module or the piezoresistive layer 300 (e.g., support for the piezoresistive elements 320 and the piezo-support 310). In some embodiments, where the lens 530 is not required, a support 800 is provided, in which the support 800 is not a lens. In some examples, the support 800 is a transparent component coupled to the housing 110, e.g., side wall of the housing 110. The support 800 is selected from a stiff material (for example but not limited to acrylic, etc.) so that it can support the piezoresistive layer in a plane normal (or substantially normal) to the primary axis 101. In some examples, the support 800 is a planar or board-like structure. In FIG. 2, the support 800 is shown disposed "on top" of the housing 110. In some other examples, the perimeter or edge of the support 800 may be engaged with the side wall of the housing 110. The support 800 enables the piezoresistive elements 320 to be supported in the space between the deformable element 600 and the camera 200, without interfering with the function of the one or more reflective elements 410.

[0040] The array of piezoresistive elements 320 is configured to collectively serve as a force sensor. Additionally, individual piezoresistive elements 320 are capable of measuring respective local forces independently of other piezoresistive elements 320 in the same array.

[0041] In some embodiments, the piezoelectric module or piezoresistive layer 300 is a multilayer sensor assembly. For example, the piezoelectric module may include multiple layers of piezoresistive elements. For example, the multilayer sensor assembly may include a first piezoresistive layer and a second piezoresistive layer. The first piezoresistive layer may be configured to sense astatic force or slowly varying forces through changes in electrical resistance. The second piezoresistive layer may be configured to sense a dynamic force or rapidly changing forces by generating a voltage in response to mechanical stress.

[0042] The first piezoresistive layer and the second piezoresistive layers may be stacked one layer on top of the other layer. For example, a first piezoelectric element and a second piezoresistive element may be stacked one on top of another to form a multilayer stack of piezoresistive elements. The piezoelectric module may include a plurality of such multilayer stacks distributed in an array on one or more piezo-supports 310. The piezoelectric module is supported by the support 800. For the sake of brevity, reference herein to "a piezoresistive layer" may be understood to refer to a piezoelectric module with the plurality of piezoresistive elements arranged in one layer or in multiple layers, with the terms "piezoelectric layer" and "piezoelectric module" used interchangeably.

[0043] Alternatively, the multilayer sensor assembly may be integrated into a single sensor unit, allowing the device 100 to capture a broad range of force magnitudes and temporal variations for enhanced tactile sensing performance.

[0044] As an example, in the prototype, the first piezoresistive layer is configured to measure or detect a high-frequency force feedback, with piezoresistive elements or force-sensitive resistors (FSR) in a four-by-four (4x4) array converting an applied pressure into a measurable resistance. The FSR was 3 mm (millimeters) in diameter. The relationship between pressure P and resistance R is given by:

[0045] where k is a material constant. As pressure increases, the resistance decreases, allowing real-time detection of applied forces.

[0046] The device 100 may include a circuit. The circuit includes a sensor acquisition circuit utilizes a virtual ground method, converting resistance to voltage. The relationship between voltage (V) and resistance (R), with a reference resistor Rref, is given by:

[0047] Given the inverse relationship between pressure and resistance, the voltage is approximately linearly proportional to pressure, allowing the expression:V oc P

[0048] This linear relationship between voltage and resistance enables precise real-time force measurement after sensor calibration.

[0049] To use the device 100, the contact end of the device 100 (or the outer surface 620 or the pigment layer 700) is brought into physical contact with the object surface of interest. In response to the physical or tactile contact between the device 100 and the object surface, the deformable element 600 may deform in shape. The deformable element 600 is made of a transparent or translucent material. A deformation in the deformable element 600 will result in a change in the way light is reflected by the pigment layer 700 and / or transmitted through the deformed deformable element 600 to reach the camera 200. The resulting image captured by the camera 200 forms at least a part of the vision-based information contributing to a tactile sensing of the object of interest.

[0050] The pigment layer 700 may be opaque, partially, or fully filled with material, patterned, or otherwise configured to achieve a desired optical effect. If the pigment 700 layer is opaque or dark-colored, external light sources are blocked from interfering with the illumination of the inner surface of the deformable element 600. In some embodiments, the pigment layer 700 includes a color that contrasts with the object of interest so as to enable formation of a high-contrast image of the visually detectable deformation. Examples of a patterned pigment layer 700 includes having a pattern of dots or markers distributed over the area of the pigment layer 700. Having dots or markers can improve the tactile sensing by presenting a lower dimension tactile representation but for the device 100 of the present disclosure, having dots or markers is optional.

[0051] According to embodiments of the present disclosure, the deformable element 600, the piezoelectric module, and the camera 200 are disposed in alignment along the primary axis 101, with the piezoresistive layer 300 disposed in between the deformable element 600 and the camera 200. To one of ordinary skill in the art, this proposed configuration would appear counter-intuitive as the opaquepiezoresistive elements 320 would block a part of the pigment layer 700 from the view of the camera 200, e.g., such a configuration would be expected to result in multiple occluded areas or blind areas 810.

[0052] The array of piezoresistive elements of the piezoresistive layer 300 is disposed between the camera and the internal surface or the internal surface of the deformable element. As the piezoresistive elements 320 are opaque, the piezoresistive elements 320 would obstruct the camera's view of the deformable element / pigment layer in various parts (the hidden or blocked areas being referred to as blind areas or occluded areas).

[0053] According to embodiments of the present disclosure, the device 100 includes a mirror system (as an example of optical elements), the mirror system being configured to provide a three-dimensional (3D) image of the pigment layer 700 / deformable element 600 in which all the blind areas are viewable by the camera via the mirror system.

[0054] Alternatively described, the optical elements enable a more complete image of the deformable element 600 / pigment layer 700 to be acquired by the camera. The more complete image can be considered equivalent to a combined view of the camera 201 (also referred to as a main camera or a primary camera to avoid confusion) and at least two virtual cameras 221 , 222. Areas blind to the view of the main camera (main camera blind areas) are viewable via the mirror system 400. The effect is equivalent to having at least two additional virtual cameras 221 ,222 on either side of the main camera, as illustrated in FIG. 2.

[0055] According to various embodiments, the force-based sensing unit and the vision-based sensing unit are simultaneously operable in response to a contact interaction between the deformable element and the object of interest. According to various embodiments, the vision-based sensing unit further includes a lighting module disposed proximally to the internal surface of the deformable element.

[0056] According to various embodiments, each mirror of the mirror system is disposed at an angle relative to the primary axis. The primary axis is a virtual line of reference coincidental with a principal optical path extending from the camera toward the deformable element. The one or more mirrors may be disposed in symmetry with respect to the optical axis. The mirrors are oriented to redirect opticalpaths from the blind areas toward the camera. The angular displacement of each of the mirrors relative to the primary axis is selected so that none of the main camera blind areas is occluded from view by the camera 200.

[0057] Referring to FIG. 2 and FIG. 3, a first lateral optical path originating from a left-side region of the deformable element 600 is redirected toward the camera 200 to form a first virtual camera view, while a second lateral optical path originating from a right-side region is redirected toward the camera 200 to form a second virtual camera view.

[0058] The device 100 may include a processor configured to stitch together the first virtual camera view, the second virtual camera view, and the main camera view, with reference to the primary axis 101. This enables the device 100 to overcome blind areas caused by the occlusion of the plurality of piezoresistive elements. The camera is enabled to produce a composite image in which there are no blind areas. In other words, the blind areas do not overlap across all the three views captured by the camera 200.

[0059] The processor may be further configured to perform camera calibration to facilitate accurate image capture. The intrinsic camera matrix used in the calibration process is:

[0060] where fxand fyrepresent the focal lengths, and cxand cyare the optical center coordinates.

[0061] Then, distortion correction is performed using the following radial and tangential distortion model:

[0062] where k ,k2,k3are the radial distortion coefficients, p ,p2are the tangential distortion coefficients, and r2= x2+ y2is the radial distance from the center of the image.

[0063] The geometry parameters are calculated, and the removal effect is validated through simulation using MATLAB as shown in FIG. 3. The labelled shapes each represent the blind areas in each of the three different camera views, respectively. The optical elements are oriented and / or disposed so that the blind areas from the three camera views have no overlap.

[0064] According to various embodiments, the device 100 includes the light guide 520 coupled to the light source 500. The light guide 510 is configured to allow light from the light source 500 to pass through at least in part. The light guide 520 may be formed from materials that are frosted, diffusive, translucent, transparent, or otherwise configured to allow light to pass through. In some embodiments, the material properties of the light guide 520 can be selected to achieve desired light distribution, diffusion, or illumination effects within the internal space of the device 100. FIG. 4 shows the improvement in the clarity of the images obtained using two versions of the device of FIG. 1B. The top image was obtained using an embodiment without the light guide ("before improvement"), and the bottom image was obtained using an embodiment with the light guide ("after improvement"). The bottom image shows a more uniform contrast between the dot markings on the pigment layer and the background. Nevertheless, it was noted that the embodiment of the top image was operational for the intended simultaneous vision-based and force-based sensing, and that the light guide 520 may therefore be optional in some of the embodiments.

[0065] According to some embodiment, the force-based sensing unit is configured to be alignable with the light guide 520 during operation. This enables the device 100 to be reconfigurable and readily assembled, allowing replacement components, such as the deformable element 600 or the plurality of piezoresistive elements 320, to be consistently aligned. The configuration ensures that both the vision-based sensing unit and force-based sensing unit remain in sufficiently precise positions during repeated use. Precise positioning maintains sensor accuracy and reliability during operation over time.

[0066] Further details of various embodiments of the device 100 will be described.

[0067] Referring again to FIG. 1A to FIG. 2, the deformable element 600 (which may also be referred to as a deformable element or an elastically deformable layer)may be colored or may have a pigment layer 700 disposed thereon. In some experiments, the deformable element 600 was made of a clear and colorless silicone or urethane rubber (e.g., Solaris available from Smooth-On, Inc.) with a pigmented layer disposed thereon. The deformable element deforms in response to an external mechanical force or pressure and is found suitable for mimicking the Slow-Adapting I (SA-I) mechanoreceptors of human skin. The deformable element 600 enables the capture of high-resolution topographical data and fine surface textures via the camera 200.

[0068] Positioned beneath the deformable element is the plurality of piezoresistive elements 320 (also referred to as force sensors). In this example, piezoresistive elements are spaced apart at 8 mm intervals along a reference direction and spaced apart at 12 mm intervals in another direction orthogonal to the reference direction. This array of force sensors is configured to detect dynamic force distributions and high-frequency tactile transients, simulating the Fast-Adapting II (FA-II) mechanoreceptors found in human skin. The piezoresistive elements 320 (the piezoresistive layer) are supported by the support 800, in which the support 800 is a non-elastomeric component offering a planar (substantially planar) surface to resist excessive displacement of the piezoresistive element. When a force is applied to particular area of the deformable element 600 / pigment layer 700, the deformation of the deformable element 600 results in a simultaneous pressure on one or more selected ones of the piezoresistive elements 320 in the particular area. Since the piezoresistive elements 320 are not attached to the deformable element, the piezoelectric elements 320 outside the particular area are not affected or are affected to a smaller degree. In this manner, a more accurate sensing of the pressure distribution is achieved and localized sensing is possible.

[0069] To mitigate optical occlusion caused by the interposition of the array of piezoresistive elements, one or more optical redirection elements (such as mirrors) is disposed within the housing 110. In this embodiment, at least two optical redirection elements are oriented at predetermined angles to provide the camera 200 with auxiliary virtual views. To facilitate mounting of the optical redirection elements or mirrors, the sidewall 140 of the housing 110 may be shaped and / or angled at a predetermined angle to the primary axis, as shown in FIG. 1A and FIG.2. These multi-view perspectives complement the camera’s direct line-of-sight, ensuring a continuous field of view across the internal surface of the deformable layer.

[0070] Illumination is managed by the light source that is coupled with the light guide. The light guide 520 may be positioned along the perimeter of the deformable element 600. The light guide ensures uniform light distribution across the inner surface to enhance image contrast. Structurally, the light guide 520 also serves as a mechanical alignment plate to facilitate precise component registration during assembly. The red-green-blue light emitting diodes 520 (RGB LEDs provide multi-spectral illumination, which creates distinct chromatic contrasts on the deformed elastomer to support depth perception and texture characterization. As illustrated in FIG. 1A, the LEDs 510 may be disposed at sides of the cover 130 or the deformable element 600.

[0071] As mentioned above, the RGB LEDs 510 set may be part of the lighting module. In some embodiments, the LEDs 510 in the device 100 are located between the pigment layer 700 and the piezoresistive layer 300 for more uniform illumination and reduced unwanted reflections. Reflective panels (or mirrors) are added on the inner side wall 140 of the housing 110, further improving light distribution and image quality. The darker image (lower image) in FIG. 4 shows the lighting conditions in a prototype of an earlier version. The brighter image (upper image) in FIG. 4 shows the lighting conditions in the device 100 according to embodiments of the present disclosure. As can be seen, adding the reflective panels made a significant improvement to the lighting conditions.

[0072] The camera 201 is mounted at the base of the housing to monitor the inner surface of the deformable element 600. The camera 201 may be equipped with a wide-angle or fisheye lens to maximize the captured area and integrate the direct and reflected views from the optical redirection elements.

[0073] As shown in FIG. 1A, this multimodal integration captures intricate tactile information, such as textures, while simultaneously measuring high-frequency force, vibration, and pressure distributions. These features enhance the potential for tasks requiring precise and responsive tactile feedback.

[0074] According to another embodiment of the present disclosure, the device 100 may have a circular configuration, as illustrated in FIG. 5A and FIG. 5B. In other embodiments, the device 100 may take alternative shapes or arrangements to suit specific operational or spatial requirements, while still incorporating the features described above. The components of the device 100 may be arranged and assembled radially which may simplify alignment. For example, the deformable element 600 may have a circular shape or a disk shape. In the case of the disk shaped elastically deformable element 660, forces applied during the contact interaction may distribute radially and evenly toward the center of the deformable element 600. The deformable element 600 may have a pigment layer 700 disposed thereon. It was found that in this embodiment, a focusing lens is optional. The cover 130 is annular. Advantageously, a more uniform illumination can be provided by having the light source 500 disposed around an inner circumferential wall of the cover 130. The piezoelectric module or the piezoresistive layer 300 includes an array of piezoresistive elements 320 disposed on a piezo-support 310. The piezoelectric module may include a frame 340. The frame 340 enables installation of the piezo-support 310 to the housing 110. In this example, the piezo-support 310 may be made of a substantially transparent or translucent material. In this example, each of the plurality of piezoresistive elements 320 or force sensors positioned beneath the deformable element 600 may be provided on a single layer to receive the sensed force proportional to the applied force. The device 100 includes a support 800 that is disposed between the piezoresistive layer and the camera 200. The support 800 provides support to the piezoresistive layer. The housing 110 may take the form of a funnel, a column, a channel, or another structure suitable for supporting and enclosing the device components. In this example, the housing 110 may be frustoconical in shape with a housing proximal end 112 that is smaller than a housing distal end 111, proximity being described relative to the base 120 for convenience. In this example, the device 100 includes a mirror system 400 that is frustoconical in shape. A mirror proximal end 402 of the mirror system 400 defines a circular cross-section that is smaller than that defined by a mirror distal end 401. In this example, the reflective element 410 is a continuous frustoconical surface that lines the inner side wall of the housing 110 to provide an all-round inclined reflectivesurface that reflects incident light towards the deformable element 600 / pigment layer 700.

[0075] FIG. 5B and FIG. 6A show the as-assembled schematic diagrams of these various embodiments of the device 100. As shown, the device 100 can have an overall compact configuration with a relatively small footprint at the base without being limited to the size of the sensing area represented by the external surface of the deformable element. This facilitates integration with various robotic applications and mobile applications.

[0076] According to yet another embodiment of the present disclosure, the deformable element 600 of the device 100 may be in the form of a hemispherical or dome shape elastically deformable element 670, as illustrated in FIG. 6A and FIG.6B. The outer surface of the hemispherical deformable element has a coating of the pigment layer 700. Similar to the other embodiments, the pigment layer 700 is optionally but not necessarily patterned with dots or markers. This advantageously simplifies the manufacturing of the device 100 as it is not required to precisely position each placement of a dot or a marker relative to the view of the camera 200. As shown FIG. 6A and FIG. 6B, the hemispherical deformable element 600 may have its flatter part in abutment with on or slightly spaced apart from the piezoresistive layer 300 (piezoelectric module), while the hemispherical part extends through the annular cover 130. The cover 130 may be engaged with the housing 110 to secure components in a stack (stacked along the primary axis 101). For example, the deformable element 600, the piezoelectric module (piezoresistive layer 300), the support 800, and the mirror system 400 (reflective element 410), and the camera 200 may be arranged in sequence as a stack.

[0077] According to various embodiments, the device 100 may be adapted for bioinspired integration, as illustrated in FIG. 7 and FIG. 8. This integration allows for the use of various deformable elements (also referred to as gel layers) and piezoresistive arrays (also referred to as force-sensing arrays) to mimic the two types of human tactile receptors, SA-I and FA-II, respectively, enabling corresponding tactile sensing functionalities. For example, the deformable element may be employed as a vision-based tactile sensing component to capture high-resolution surface texture information, mimicking the human SA-I receptor cell. TheSA-I cell exhibits a high spatial resolution of approximately 0.5 mm and operates in a low-frequency range of about 0.4 to 3.5 Hz, making it sensitive to low-frequency and dense tactile stimuli. Accordingly, the SA-I cell is sensitive to spatial deformation and surface features, enabling texture discrimination and contact localization.

[0078] Similarly, the plurality of force sensors arranged in a customizable forcesensing array can emulate FA-II receptor cells. The FA-II cell exhibits a low spatial resolution of more than 10 mm and operates in a high-frequency range of 40 to 500 Hertz (Hz), making it sensitive to dynamic stimuli. Accordingly, the FA-II cell is sensitive to temporal force changing and capable of vibration sensing, tool manipulation, and slip detection. The flexibility of this integration to mimic both the SA-I and FA-II cells allow for optimization based on specific task requirements. In one embodiment, the device 100 may be configured to operate with a high resolution of less than 0.1 mm which corresponds to 560 x 450 pixels for a 41 x 33 mm area, and a low frequency range of 0 to 20 Hz.

[0079] Experimental Validation

[0080] Prototypes of the device 100 were fabricated and used in several experiments to validate the viability of the device 100. The prototype of device 100 incorporated a 4 x 4 piezoresistive array and a planar gel layer (deformable element). The piezoresistive array includes 16 force sense resistors (FSRs). The experiments demonstrated the capability of the device 100 to achieve a resolution of 560 x 450 pixels and a response frequency of up to 300 Hz, among other performance metrics.

[0081] For normal pressure measurement, the piezoresistive layer was calibrated to determine the relationship between the voltage converted from the piezoresistive layer’s resistance change via the acquisition circuit and the actual applied force. Hysteresis and linearity tests are also performed. Since all 16 FSRs are identical, one FSR is calibrated, and its parameters were used to calibrate the others.

[0082] FIG. 9 illustrates the experimental setup of the device 100 in which a force gauge probe 914 is placed vertically over the gel layer (deformable element). Top view of the experimental setup showed the probe’s circular cross-section to beconcentric with the corresponding FSR’s sensing area beneath the gel layer. The probe had a radius that is slightly larger than that of the FSR, such that uniform pressure is applied to the FSR. The pressure measurement test is divided into two stages: a loading phase and an unloading phase.

[0083] In the loading phase, the force gauge probe is gradually pressed onto the gel layer with one FSR placed beneath. Using a KUKA iiwa robot manipulator (KUKA LBR iiwa robotic arm 912 and Robotiq Gripper end effector 910), the force gauge is lowered by 0.1 mm with each step. After a 5-second steady period, both the force applied by the force gauge and the voltage collected by the acquisition circuit were recorded. This process is repeated until the output voltage reaches its maximum of 5 V.

[0084] In the unloading phase, the force gauge is lifted with the same step as used in the loading phase, and the applied pressure is reduced until the force returns to zero, with data recorded throughout the process.

[0085] High Frequency Vibration Response Test

[0086] To validate the FSRs ability to detect high-frequency signals, a high frequency response test for a prototype of the device 100 is conducted using a Faulhaber LM1483 linear motor 920 to generate periodic displacements at 25 Hertz (Hz) and 50 Hz. In the experimental setup as shown in FIG. 10, the motor’s moving probe is positioned vertically above the gel layer. Top view of the experimental setup showed that the probe’s circular cross-section is concentric with the corresponding FSR’s sensing area beneath the gel layer. The probe had a radius that is slightly larger than that of the FSR, such that uniform pressure is applied to the FSR, consistent with the pressure measurement test.

[0087] To investigate the damping effect of the gel layer with a thickness of 8 mm on high-frequency signals, an additional FSR is placed on the gel layer’s surface to measure the ground truth force at the surface. By comparing the amplitude of the forces recorded by both FSRs, the amplitude attenuation rate can be calculated. Based on the force calibration data, the forces exerted by the motor’s periodic motion at 25 Hz and 50 Hz for both FSRs are recorded, respectively.

[0088] In-Sensor Occlusion Removal Test

[0089] The camera and the mirror system capture the deformation of the gel layer from three different angles: L (left), M (main), and R (right). The main view (M) serves as the primary perspective, while the left and right views supplement the occluded areas in the main view. The three views are stitched together to generate tactile maps comprising visual representations of the deformation from multiple sensor readings using an image stitching algorithm.

[0090] To validate the effectiveness of the stitching algorithm, an experiment is conducted by pressing a piece of dental floss onto the surface of the device 100 (FIG. 9). As shown in FIG. 11, a series of image processing methods are used to retrieve the texture data of the dental floss sample.

[0091] As illustrated by the flowchart diagram in FIG. 11, the occlusion removal test began by using a dental floss engraved with letters as a test object. The test object was then pressed onto the testing setup. Next, the fisheye camera captured the deformation of the gel from multiple angles with the reflective mirror system. Next, the camera distortion in the raw image captured was corrected and calibrated, then the calibrated image was segmented and split into the three views, left, right and main. Lastly, by using OpenCV, the occluded areas in the main view caused by the piezoresistive layer can be removed, and the information captured from the left and right views are used to fill in these blind spots as shown in the processed result.

[0092] Results on Force Calibration

[0093] The measured voltage and resistance are shown in FIG. 12. In this experiment, a virtual ground method for the acquisition circuit design is used. Theoretically, the converted voltage should be approximately linearly related to the FSR’s conductance within the FSR’s measurement range. Since the FSR’s conductance is also linearly related to the applied pressure, a linear fitting is applied to obtain the sensor’s voltage-force relationship. FIG. 12 illustrates the results and relationship between that of the force calibration, hysteresis precision, and linearity results.

[0094] The force calibration experiment revealed a linear correlation between the measured voltage and the applied force. The results shown in FIG. 12 indicated that the sensor could detect normal pressures applied on its surface. The maximumhysteresis error was measured to be 0.395 V, and the linearity error was 14.01%. The device demonstrated significant potential for practical applications.

[0095] Results on High-Frequency Vibration Response

[0096] The sensor’s ability to detect high-frequency signals was verified through the vibration experiments at 25 Hz and 50 Hz. The time-position curve of the motor, and the force curves recorded by the surface FSR, and the sensor FSR are shown in FIG. 13A to FIG. 13D, respectively. The sensor can accurately capture dynamic force inputs, with attenuation rates of 9.71% for 25 Hz and 11.64% for 50 Hz signals. These results validated the sensor’s capability to handle high-frequency tactile information, making the sensor suitable for dynamic manipulation tasks requiring precise and real-time feedback.

[0097] The occlusion removal experiment using the reflective mirror system and the fisheye camera was carried out to demonstrate the sensor’s ability to effectively remove occluded areas in the main view. FIG. 14A shows the result of using the algorithm to remove the occlusion caused by the piezoresistive layer, while FIG.14B shows the image presenting the ground truth texture obtained by physically removing the piezoresistive layer from the structure. This comparison confirmed that the device 100, in combination with its stitching algorithm, can effectively reconstruct and integrate multimodal information from multiple views, thereby significantly enhancing texture perception in occluded regions.

[0098] Results on Positional Error and Force Limit

[0099] Positional deviations were observed in the markers within the processed result after applying the stitching algorithm. The average positional error of the markers was calculated under a no-input condition. The calculation method is illustrated in FIG. 15, where labelled circles indicate the marker positions as the ground truth captured by physically removing the piezoresistive layer or marker positions obtained through our image processing algorithm after occlusion removal. In the non-occluded regions, the marker positions remained consistent before and after processing, aligning with the ground truth. In the occluded regions, processed marker coordinates exhibited a measurable offset compared to the ground truth. The average positional error in the x and y directions was computed using:

[0100] where N represents the total number of markers in the occluded area, and Axtand Aytare the individual positional deviations in the x and y directions.

[0101] Excessive deformation of the gel layer can cause issues in the image stitching algorithm, resulting in texture distortion and the markers losing track. Therefore, the sensor’s force limit was measured to determine the maximum normal pressure under which all markers remain identifiable. Force limit measurements were conducted in the four stitching regions of the sensor, and current results show that the sensor can maintain all markers visible and accurately detectable up to a maximum normal force of 10.2 N, with a corresponding vertical deformation of 2.4 mm.

[0102] To further investigate the effect of gel deformation depth on marker detection, an experiment is conducted to measure the deviation between the processed markers and the ground truth under varying compression depths. The experimental setup was the same as that shown in FIGS. 8 and 9. The gel surface was compressed to different depths at 16 different regions on the sensor. To ensure no marker loss during tracking, the compression depth was limited to half of the maximum measurable deformation range, i.e., 0 mm to 1.2 mm. For each compression depth, processed images were obtained using the stitching algorithm, and marker positions were recorded. The experiment was then repeated after physically removing the piezoresistive layer, with identical compression settings. The second set of data, without occlusion, was used as the ground truth.

[0103] The root mean square error (RMSE) of the marker coordinates (in pixels) between the processed result and the ground truth was used as the evaluation metric for the algorithm’s accuracy. FIG. 16 shows the relationship between the x and y coordinate deviations and the gel compression depth in terms of RMSE and press depth.

[0104] Both x and y directional deviations increased with the compression depth, reaching average deviations of 1.26 pixels and 0.84 pixels, respectively, at a maximum depth of 1.2 mm. The relationship between pixel deviation andcompression depth is approximately linear, attributed to the planar assumption of the gel in the stitching algorithm. The results in the x-direction were more significantly affected due to the placement of the reflective mirror system along the x-axis, amplifying the error in this direction.

[0105] A comparison with several conventional tactile sensors is provided. The comparison is based on key criteria including the sensing modalities, frequency response, density resolution, and functionalities such as texture recognition, high-frequency vibration detection, and pressure distribution. The comparison analysis is listed in Table 1 below. The sensor named "MultiTac" is a prototype of the proposed device 100.

[0106] Table 1: Comparison of the experimental results of the device 100 with other tactile sensors

[0107] In evaluating the measurement range, the comparison considered the saturation or accuracy limit of each sensor’s modalities. Specifically, for each sensor, the range presented was based on the lowest measurement range among all modalities where saturation or precision loss occurs. In prototype of the device 100 used in the experiments, the piezoresistive layer had a range of 0 to 8 N, andthe vision-based modality supported a range up to 10 N, A pressure range of 0 to 8 N was selected as the effective pressure range for the device 100 for the sake of this comparison. Advantageously, the pressure sensing range of the device 100 is customizable.

[0108] Table 2: Comparison of applications of the device 100 with other tactile sensors. In the table, the sensor named "MultiTac" refers to the proposed device 100.

[0109] The experimental data verified that the device 100 outperforms conventional sensors, enabling both high-frequency and high-resolution tactile sensing. It also supports texture recognition and accurate pressure distribution, making it suitable for dynamic dexterous manipulation tasks. The experiments demonstrated that the device 100 is capable of high frequency sensing, e.g., at least 300 Hz or higher. It is also possible to have a high frequency sensing as high as 1000 Hz. The device 100 is operational at a high frequency sensing range, for example but not limited to a range from 300 Hz to 1000 Hz. FIG. 17 illustrates a use case of the device 100 in tactile servoing, in which the device 100 manipulates deformable objects 900 during operations such as cable manipulation and cable or plug insertion. During such operations, the device may be coupled with additional components, including, but not limited to, grippers 910, robotic arm manipulators, end-effectors, or other actuation mechanisms. The additional components may be configured to provide controlled positioning, orientation, or force application, while the device 100 provides tactile feedback and sensing to guide the manipulation. Insome embodiments, the grippers or manipulators may be interchangeable or adjustable to accommodate objects of varying size, shape, or compliance, enabling the system to adapt to diverse operational requirements.

[0110] The present application discloses the device 100 for multimodal tactile sensing and the integration of high-resolution vision-based sensing with high-frequency force sensing. The device 100 has flexible configurations, including variations in the arrangement and dimensions of the gel layer and piezoresistive array, to suit diverse applications.

[0111] In experiment, the prototype of the device 100 achieved a sensing spatial resolution equivalent to 560x450 pixels and a temporal response rate of at least 300 Hz. Higher sending resolution is achievable with the device 100, i.e. , the device may be configured to have a sensing spatial resolution equivalent to 560 pixels by 450 pixels, or higher. These results demonstrated the potential of the proposed integration method to provide precise and responsive tactile feedback, making it suitable for complex dynamic tasks that require both fine surface detail and high-frequency force sensing. Additionally, the device 100’s prototype incorporated a mirror system to minimize interference between sensing modalities and a reconfigurable structure to maintain alignment during repeated assembly.

[0112] The placement of the piezoresistive elements 320 at the inner surface of the deformable element 600 advantageously enables the deformable element to deform while the piezoresistive elements 320 remain fixed in position relative to the camera 200. This enables the same external force or pressure (from contact interaction between the outer surface of the deformable element 600 and the object of interest) to simultaneously deform the deformable element 600 and cause a change in resistance / voltage in the piezoresistive layer 300. The result is a realtime and simultaneous response in more than one mode. Measurements are taken simultaneously in different ways based on the same instantaneous input of a force or pressure in one local area, such that the sensing can better correspond to human tactile sensing.

[0113] In another aspect, the piezoresistive layer is advantageously protected by the deformable element. The positions of the piezoresistive elements are fixed, and the piezoresistive elements would be less prone to damage as non-movingcomponents are generally less likely to suffer as much wear-and-tear as moving components.

[0114] According to various embodiments of the present disclosure, a device or a sensor device includes a housing, a deformable element, a camera, and a piezoresistive layer. The housing has a side wall extending between a base and a cover. The base and the cover are spaced apart from one another along a primary axis defined by the housing. The deformable element is disposed at the cover. The camera is disposed at the base. The camera is oriented with its optical axis coincidental with the primary axis. Deformation of the deformable element is visually detectable by the camera. The piezoresistive layer includes a plurality of piezoresistive elements disposed in an array, in which the piezoresistive layer is disposed at the inner surface of the deformable element between the deformable element and the camera.

[0115] The device may further include a support. The support is a transparent component coupled to the side wall. The piezoresistive layer is supported by the support.

[0116] The device further includes one or more reflective elements disposed at the side wall. The reflective elements are angularly displaced relative to the primary axis to reflect light from the deformable element to the camera.

[0117] The device further includes a pigment layer disposed at an outer surface of the deformable element, the deformable layer being transparent or translucent, in which changes in the pigment layer is visually detectable by the camera.

[0118] Optionally, the device further include a light guide. The light guide may be disposed at a perimeter of the deformable element.

[0119] The device may further include a lighting module coupled to the cover. The lighting module is configured to illuminate the deformable element.

[0120] The device may further include: a processor configured to execute instructions stored in a machine-readable memory to perform a method, in which the method includes: capturing a plurality of images of the deformable element, at a same time instant, each of the plurality of images including a plurality of blind areas, each blind area corresponding to an area of the deformable element that is blocked from the camera view by one or more of the plurality of piezoresistiveelements; and combining the plurality of images to obtain a combined image with no blind areas.

[0121] The device may further include: two reflective elements disposed at the side wall, the reflective elements being angularly displaced relative to the primary axis to reflect light from the deformable element to the camera; a processor, the processor being configured to execute instructions stored in a machine-readable memory to perform a method, the method including: capturing a main camera view, a first virtual camera view, and a second virtual camera view, the plurality of piezoresistive elements forming a plurality of main blind areas in the main camera view, a plurality of first blind areas in the first virtual camera view, and a plurality of second blind areas in the second virtual cameral view; and combining the main camera view, the first virtual camera view, and the second virtual camera view, wherein none of the plurality of main blind areas overlaps with any one of the first blind areas and the second blind areas.

[0122] The device may be configured to have a sensing spatial resolution equivalent to 560 pixels by 450 pixels, or higher.

[0123] The device may be configured to have a frequency sensing range, in which the frequency sensing range has an upper limit from 300 Hz to 1000 Hz. The device may be configured to have a customizable pressure sensing range. In some applications, the device is capable of a frequency sensing range of up to 1000 Hz. In some applications, the device is capable of a frequency sensing range of up to 300 Hz.

[0124] The device may be configured to have sensing modalities including visionbased, force-based, and vibration-based sensing.

[0125] The deformable element may be any one of a dome shape, a circular shape, and a rectangular shape.

[0126] In another aspect, a device for multimodal tactile sensing includes: a housing, a vision-based sensing unit, and a force-based sensing unit. The housing has a side wall extending between a base and a cover, the base and the cover being spaced apart from one another along a primary axis defined by the housing. The vision-based sensing unit includes: a deformable element, the deformable element being disposed at the cover, the deformable element being elasticallydeformable and transparent; a camera, the camera being disposed at the base, the camera being oriented with its optical axis coincidental with the primary axis, deformation of the deformable element being visually detectable by the camera. The force-based sensing unit includes: a piezoresistive layer, the piezoresistive layer including a plurality of piezoresistive elements disposed in an array, wherein the piezoresistive layer is disposed at the inner surface of the deformable element between the deformable element and the camera.

[0127] The device may further include a support, the support being a transparent component coupled to the side wall, the piezoresistive layer being supported by the support.

[0128] The device may further include one or more reflective elements, the one or more reflective elements being disposed at a perimeter of the piezoresistive layer, the camera being configured to capture parts of an image of an inner surface of the deformable element occluded by the plurality of piezoresistive elements.

[0129] The force-based sensing unit and the vision-based sensing unit may be configured to be simultaneously operable in response to a same force or pressure at an outer surface of the deformable element.

[0130] The vision-based sensing unit may include a lighting module disposed proximal to an inner surface of the deformable element.

[0131] All examples described herein, whether of apparatus, methods, materials, or products, are presented for the purpose of illustration and to aid understanding and are not intended to be limiting or exhaustive. Modifications not involving inventive effort may be made by one of ordinary skill in the art without departing from the scope of the claimed invention.

Claims

CLAIMS1. A device comprising:a housing, the housing having a side wall extending between a base and a cover, the base and the cover being spaced apart from one another along a primary axis defined by the housing;a deformable element, the deformable element being disposed at the cover; a camera, the camera being disposed at the base, the camera being oriented with its optical axis coincidental with the primary axis, deformation of the deformable element being visually detectable by the camera; and a piezoresistive layer, the piezoresistive layer including a plurality of piezoresistive elements disposed in an array,wherein the piezoresistive layer is disposed at the inner surface of the deformable element between the deformable element and the camera.

2. The device as recited in claim 1 , further comprising a support, the support being a transparent component coupled to the side wall, the piezoresistive layer being supported by the support.

3. The device as recited in claim 1 or claim 2, further comprising one or more reflective elements disposed at the side wall, the reflective elements being angularly displaced relative to the primary axis to reflect light from the deformable element to the camera.

4. The device as recited in claim 3, further comprising a pigment layer disposed at an outer surface of the deformable element, the deformable layer being transparent or translucent, wherein changes in the pigment layer is visually detectable by the camera.

5. The device as recited in any one of claims 1 to 4, further comprising:a light guide, the light guide being disposed at a perimeter of the deformable element.

6. The device as recited in any one of claims 1 to 5, further comprising a lighting module coupled to the cover, the lighting module being configured to illuminate the deformable element.

7. The device as recited in claim 3, further comprising: a processor, the processor being configured to execute instructions stored in a machine-readable memory to perform a method, the method including:capturing a plurality of images of the deformable element, at a same time instant, each of the plurality of images including a plurality of blind areas, each blind area corresponding to an area of the deformable element that is blocked from the camera view by one or more of the plurality of piezoresistive elements; andcombining the plurality of images to obtain a combined image with no blind areas.

8. The device as recited in claim 1 , further comprising:two reflective elements disposed at the side wall, the reflective elements being angularly displaced relative to the primary axis to reflect light from the deformable element to the camera.a processor, the processor being configured to execute instructions stored in a machine-readable memory to perform a method, the method including:capturing a main camera view, a first virtual camera view, and a second virtual camera view, the plurality of piezoresistive elements forming a plurality of main blind areas in the main camera view, a plurality of first blind areas in the first virtual camera view, and a plurality of second blind areas in the second virtual camera view; andcombining the main camera view, the first virtual camera view, and the second virtual camera view,wherein none of the plurality of main blind areas overlaps with any one of the first blind areas and the second blind areas.

9. The device as recited in any one of the preceding claims, wherein the device is configured to have a sensing spatial resolution equivalent to 560 pixels by 450 pixels, or higher.

10. The device as recited in claim 1, wherein the device has a frequency sensing range having an upper limit from 300 Hz to 1000 Hz, and a customizable pressure sensing range.

11. The device as recited in claim 1 , wherein the device is configured to have sensing modalities including vision-based, force-based, and vibration-based sensing.

12. The device as recited in claim 1 , wherein the deformable element is any one of a dome shape, a circular shape, and a rectangular shape.

13. A device for multimodal tactile sensing, comprising:a housing, the housing having a side wall extending between a base and a cover, the base and the cover being spaced apart from one another along a primary axis defined by the housing;a vision-based sensing unit, the vision-based sensing unit including:a deformable element, the deformable element being disposed at the cover, the deformable element being elastically deformable and transparent;a camera, the camera being disposed at the base, the camera being oriented with its optical axis coincidental with the primary axis, deformation of the deformable element being visually detectable by the camera; anda force-based sensing unit, the force-based sensing unit including:a piezoresistive layer, the piezoresistive layer including a plurality of piezoresistive elements disposed in an array,wherein the piezoresistive layer is disposed at the inner surface of the deformable element between the deformable element and the camera.

14. The device as recited in claim 13, further comprising a support, the support being a transparent component coupled to the side wall, the piezoresistive layer being supported by the support.

15. The device as recited in claim 13 or claim 14, further comprising one or more reflective elements, the one or more reflective elements being disposed at a perimeter of the piezoresistive layer, the camera being configured to capture parts of an image of an inner surface of the deformable element occluded by the plurality of piezoresistive elements.

16. The device as recited in any one of claims 13 to 15, wherein the force-based sensing unit and the vision-based sensing unit are simultaneously operable in response to a same force or pressure at an outer surface of the deformable element.

17. The device as recited in any one of claims 13 to 15, wherein the vision-based sensing unit comprises a lighting module disposed proximal to an inner surface of the deformable element.

Images