373 results about "Tactile perception" patented technology

The invention provides an industrial robot autonomous collaborative decision-making method and system based on multi-modal sensing and a medium, and belongs to the technical field of industrial robot intelligent control. The method comprises the steps of performing cross-modal space-time alignment to eliminate data space-time differences by collecting visual, tactile and auditory information, realizing multi-modal feature fusion in combination with a dynamic weight adjustment mechanism and an attention calculation model, and generating a joint decision strategy through a deep learning optimization model. The system dynamically allocates sensor weights according to task types, introduces a multi-objective optimization mechanism of energy consumption, precision and safety, and sets a fault-tolerant rule to automatically recover the weights or recalibrate the sensors. According to the method, the problems of rigid data fusion and single optimization dimension in traditional multi-modal decision making are solved, and the precision, the response speed and the environmental adaptability of collaborative operation of the industrial robot are remarkably improved.

The invention relates to the technical field of body-equipped intelligent robots, in particular to a multi-mode body-equipped intelligent robot control method and device, and the method comprises the steps: synchronously collecting visual, auditory, tactile, force sense and body perception information, and unifying the information to the same time-space reference through a cross-mode time-space stamp alignment mechanism; hierarchical feature extraction and fusion are carried out on the multi-modal information, and unified multi-modal scene state representation is generated; reasoning a decision based on the representation by using a body agent framework, and outputting a control instruction; motion planning and control, visual servo tracking in a non-contact stage and dynamic parameter correction in a contact stage are executed according to instructions; optimizing the multi-modal strategy network through an incremental strategy distillation mechanism based on the interactive data flow; the problem of space-time asynchronization of multi-modal sensing information is solved through a cross-modal space-time stamp alignment mechanism.

The invention provides an intelligent grabbing control method and system based on tactile perception, and the method comprises the steps: obtaining a tactile feedback signal through a sensor array of a humanoid robot, carrying out the frequency domain decomposition of the signal through a Fourier transform method, and carrying out the analysis of the pressure and deformation data of each contact point on the surface of a to-be-grabbed object, calculating to obtain frequency characteristic distribution of the tactile feedback signal; if the initial estimation value of the object rigidity gradient exceeds a threshold value, performing finite element analysis on the object rigidity gradient, and combining a time domain processing result of the tactile feedback signal to obtain object rigidity gradient distribution; carrying out probability modeling on the deformation field by adopting a Bayesian inference method according to the gradient distribution of the rigidity of the object, and carrying out iterative optimization on a modeling result to obtain dynamic update parameters mapped by the deformation field; and regenerating an action sequence of the end effector of the humanoid robot based on the optimized strategy according to a trigger condition of grabbing failure detection, and determining a final grabbing control parameter.

The invention discloses a manipulator grabbing control system based on visual identification, and the system comprises a visual identification module which is used for collecting the three-dimensional form data and surface feature information of a target object, and eliminating the noise interference in an original image through preprocessing; the multi-modal fusion processor is used for receiving the three-dimensional form data and the surface feature information output by the visual recognition module, executing point cloud structured conversion on the three-dimensional form data, extracting geometric features of an object through a neural network, and meanwhile, performing weight distribution on the surface feature information by adopting a feature attention mechanism to complete multi-feature fusion processing; according to the multi-axis cooperative manipulator provided with the end executor, the end executor integrates a touch sensing assembly used for touch sensing and an adsorption execution device used for assisting grabbing. By integrating the technologies of visual identification, multi-modal fusion, intelligent optimization, dynamic feedback and the like, the sensing precision, the environmental adaptability and the grabbing reliability of the system to object characteristics are improved.

The present disclosure describes systems, devices, and methods of benefiting a cognitive state or function of a brain of a subject. The systems can include a portable system comprising a single headset device to be worn by a subject, comprising a pair of glasses configured to provide a visual stimulus to the subject, and a pair of supra-aural headphones configured to provide an auditory stimulus to the subject.

The invention discloses a five-finger bionic dexterous hand with distributed touch perception and a control method. The five-finger bionic dexterous hand comprises a palm part; fingers of the thumb unit, the index finger unit, the middle finger unit, the ring finger unit and the little finger unit respectively comprise a three-stage series flexion and extension mechanism which is sequentially composed of a third knuckle, a second knuckle and a first knuckle from a metacarpophalangeal joint, and each three-stage series flexion and extension mechanism has three flexion and extension freedom degrees and one deflection freedom degree; the tendon rope driving units are arranged in the palm part; the palm center camera is integrated on the front face of the palm part and used for collecting visual information of an operation area; the 15 air pressure type touch sensor modules are installed on the third knuckles, the second knuckles and the first knuckles of all the finger units in a distributed mode, and each air pressure type touch sensor module is integrated with a pressure and temperature double sensing unit and used for sensing contact pressure and environment temperature; the control module is integrated on the back face of the palm part and is in signal connection with the palm center camera, the air pressure type touch sensor modules and the tendon rope driving unit.

The invention relates to the technical field of voice processing, can be applied to business scenes of financial science and technology, medical health and the like, and discloses a voice text bidirectional conversion method, device, equipment and medium, and the method comprises the steps: respectively executing voice recognition or voice synthesis operation according to the type of input information; for the voice information, noise suppression parameters are generated in combination with the lip movement video data, noise reduction processing is executed, and the recognition accuracy is improved; for text information, a pre-generated speaker style vector is obtained, the vector is cited in the speech synthesis process to generate natural personalized speech, and lip movement information and tactile feedback which are synchronous with speech output are generated. According to the method, complex noise is suppressed by fusing lip movement data, personalized voice is generated by using the style vector, and lip movement and touch information is output, so that bidirectional real-time conversion of voice and text in a complex environment is realized, and recognition accuracy, voice naturalness and interaction synchronism are effectively improved.

The invention discloses a cavity environment real-time monitoring method based on multi-mode perception and a related device. The method comprises the following steps: synchronously acquiring cavity vision, contact force and electromagnetic positioning data; respectively refining the three types of data; extracting YOLO features from the refined visual data, outputting dynamic tactile features from the contact force data by using a mamba model, and refining the positioning data into positioning features through extended Kalman filtering; respectively generating classification results based on the three types of features, fusing through weighted voting, and mapping the visual and tactile features to a unified three-dimensional coordinate for association marking; and finally, generating an environment state decision according to judgment and association results. According to the invention, multi-mode real-time perception and risk early warning of the cavity environment are realized, and the operation safety and efficiency are improved.

The invention provides a pod pepper picking attitude estimation method based on visual and tactile combined perception, relates to the technical field of agricultural product picking attitude estimation, and remarkably improves the precision and adaptability of pod pepper picking attitude estimation through dynamic fusion of visual and tactile data and six-degree-of-freedom pose collaborative calculation. Based on weighted harmony of a gradient magnitude mean value of a pepper body mask boundary and a depth map shielding rate, visual confidence is generated, a tactile pressure mean value, a variance and a friction force fluctuation coefficient are combined with tactile confidence, and a normalized exponential function is utilized to dynamically distribute weights, so that when a pepper body is shielded by branches and leaves and visual features are fuzzy, the visual feature is fuzzy. The touch confidence coefficient weight is improved, the visual weight is enhanced when a touch signal is interfered by mechanical vibration, self-adaptive complementation of multi-modal data is achieved, the sensitivity of vision to pepper body contour deformation is enhanced by calculating the six-degree-of-freedom poses of vision and touch, and the judgment capacity of touch to the contact state is improved.

According to the BIM-based construction process monitoring and early warning system provided by the invention, the data acquisition dimension and precision are remarkably improved through multi-source sensing fusion of millimeter-wave radar, multispectral imaging and voiceprint recognition; feature vector voxel units carrying material characteristics and process constraints are adopted, so that risk early warning has space-time relevance and process interpretability; through a composite risk calculation model containing environmental interference correction and time-varying gradient, the problem that a traditional threshold value method is poor in adaptability to complex working conditions is solved; the holographic early warning mechanism realizes upgrading from sound-light alarm to touch sense-three-dimensional projection cooperative interaction; the double-circulation self-optimization system not only guarantees real-time control, but also realizes block chain evidence storage of empirical data, and finally forms a construction monitoring closed-loop system with space-time perception, intelligent decision, accurate early warning and sustainable evolution capabilities.

The invention relates to the technical field of medical education, in particular to a virtual patient skin care teaching system with enhanced pathological features and a method thereof, the system comprises pathological feature generation, parameter control, physical modeling, intelligent evaluation and a multi-mode feedback module, and can edit and simulate various skin pathological features in real time, such as pressure sores and diabetic feet, so that the system can be applied to the field of medical education. The visual display of the lesion degree is realized through a five-segment color mapping system, the system also simulates percolate reflection and cuticle spalling, the sense of reality is enhanced, the intelligent assessment module provides risk assessment and healing prediction according to physiological and morphological parameters, the multi-modal feedback module integrates visual sense, tactile sense and user behavior feedback, the interactive experience is improved, and the system is suitable for popularization and application. And the adaptive optimization module dynamically adjusts system parameters according to user feedback and learning effects, and provides personalized teaching. Compared with a traditional method, the method is higher in parameter control precision, supports personalized teaching, and is of great significance to medical education.

The utility model discloses a multi-mode fusion touch fingertip module which comprises a supporting layer (107). The outer surface of the supporting layer (107) is provided with a proximity sense sensing layer (101) used for sensing the distance, a temperature site (102) used for sensing the temperature, a touch sense / slip sense sensing layer (103) used for sensing the touch sense, a control element (104), an electrode layer (105) used for transmitting signals and a force sensing layer (106) used for sensing the force. According to a three-dimensional force three-dimensional tactile sensing network, multi-mode sensor signals can be obtained, a multi-branch tactile neural network is constructed by using the signals, and a tactile fingertip module is formed, so that the shortages of two-dimensional tactile sensing can be made up, and three-dimensional characteristics of tactile sensing can be obtained; that is to say, richer haptic information such as output magnitude, direction, acting position, spatial distribution of force, dynamic force time sequence characteristics, spatial distribution of temperature and the like can be decoupled, various characteristics of the object are extracted, and the accuracy of object haptic recognition is improved.

The invention relates to a soft mesh magnetic sensor for tactile and non-tactile perception. The sensor comprises a neodymium iron boron-organic silicon elastomer magnetic film, an aluminum foil, a PDMS elastomer, a TMR element and a printed circuit board, wherein the TMR element is welded on the printed circuit board, and the PDMS elastomer coats the TMR element and covers the PCB; the PDMS elastomer is covered with an aluminum foil, and a neodymium iron boron-organic silicon elastomer magnetic film is arranged on the aluminum foil; the PDMS elastomer is of a sandwich structure and is composed of an upper thin film, a lower thin film and a middle small cylinder array. The lower film is provided with a strip-shaped groove which is used for wrapping the TMR element. The soft grid magnetic sensor disclosed by the invention has the advantages of high sensitivity, high flexibility and strong multi-mode sensing capability.

The invention provides a total sensory data processing method and device based on AR glasses and electronic equipment, and relates to the field of data processing. In the method, total sensory data for a target environment sent by a multi-modal sensor is obtained, and the multi-modal sensor is located on AR glasses; performing feature extraction on the total sensory data by adopting a multi-modal dynamic fusion network to obtain a visual feature, an auditory feature, an olfactory feature and a tactile feature; determining a synchronization event from the visual feature, the auditory feature, the olfactory feature and the tactile feature through a time alignment model; and according to the synchronization event, generating a multi-modal feedback strategy, and controlling the AR glasses to execute the multi-modal feedback strategy. By implementing the technical scheme provided by the invention, the use experience of a wearer of the AR glasses can be improved.

The invention discloses a spine tactile sensing system and method based on a multi-modal sensor, and the method comprises the steps: obtaining an original tactile signal, carrying out the preprocessing of the original tactile signal, carrying out the feature extraction based on the preprocessed original tactile signal, and obtaining a spatial distribution feature and a time dynamic feature; on the basis of tail end coordinates collected by forward kinematics of the tail end of the mechanical arm, space mapping of the tactile signals and the spine anatomical structure is constructed, and anatomical position features are obtained on the basis of the preprocessed original tactile signals and the space mapping; constructing task-oriented feature vectors based on the spatial distribution features, the anatomical position features and the time dynamic features, wherein the task-oriented feature vectors comprise a sliding state recognition feature vector and a spine ordinal number positioning feature vector; and constructing a multi-stage adaptive evolution hidden Markov model, and realizing spine ordinal number positioning in combination with the task-oriented feature vector. According to the method, through multi-dimensional feature fusion and dynamic model adjustment, anatomical prior knowledge and real-time tactile signals are combined, and spine segments are accurately recognized.

The invention provides an underwater static pressure self-balancing visual touch perception device and method. The device comprises an elastic body, a hexagonal tray, a light source, a camera, a protective cover, a base, a water inlet pipe, a water outlet pipe and a computer. The elastomer is used for contacting an object, and a plurality of micron-sized through holes for balancing internal and external water pressure are formed in the surface; the elastomer is arranged on the inner side of the hexagonal tray, and the RGB three-color light source is arranged on the outer side to realize uniform distribution of light; the camera collects a light intensity image reflected by the elastomer; the protective cover is fixedly connected to the base and used for isolating external light and fixing the hexagonal tray. A camera is mounted on the base; the computer is used to model the acquired images or videos as different haptic features.

The invention discloses a mechanical arm autonomous grabbing system and method based on visual guidance and visual touch fusion. The mechanical arm autonomous grabbing system at least comprises a visual semantic guidance and grabbing pose generation module, a smooth grabbing control module based on visual touch fusion and a dynamic planning module based on model prediction path integration. The visual module efficiently generates a six-degree-of-freedom grabbing pose from the point cloud data by using a deep learning network; the smooth grabbing control module monitors object slippage in real time and adaptively adjusts clamping force by fusing visual and tactile information, and stable grabbing of fragile or slippery objects is achieved. And the dynamic planning module adopts a model prediction path integral algorithm MPPI to realize collision-free path planning of the mechanical arm in a complex dynamic environment. The problems that an existing mechanical arm is insufficient in sensing capacity, poor in grabbing robustness and low in path planning efficiency in an unstructured scene are solved, and the autonomous operation capacity and safety of the mechanical arm in the scenes of accompanying, intelligent sorting and the like are remarkably improved.

The invention discloses a multi-mode perception collaboration and real-time synchronization system, which breaks through the limitation of vision / hearing in the prior art, covers vision, hearing, smell and movement, and realizes more accurate cooperation through a perception layer, a control layer and an execution layer. According to the invention, an immersive experience space full of shock and reality can be created, synchronous output of visual, auditory, tactile and olfactory signals is realized by means of high-precision equipment, and a series of cross-sensory signal superposition schemes are designed on the basis of response characteristics of human physiology and psychology to various sensory stimuli. Therefore, the overall perception of the experiencer to the space atmosphere is enhanced. Meanwhile, through weight distribution and conflict resolution, the system can flexibly adjust the sensing mode according to the environment and the user state. The multi-device cooperation error can be controlled at millisecond level and millimeter level, and the natural fusion of virtuality and reality is realized.

The invention relates to a personalized exercise rehabilitation risk early warning method and device based on IMU dynamic instability assessment, and the method comprises the steps: collecting multi-dimensional exercise data of a subject, carrying out the posture fusion, and obtaining the limb posture, joint angle change rate and angular momentum of the subject; performing time sequence mode decomposition to extract a single gait cycle and a specific jump landing attitude; calculating a biomechanical feature vector of each motion primitive; inputting the single gait cycle, the specific jump landing attitude and the biomechanical feature vector into a hybrid injury induced motion model, and outputting a multi-dimensional injury risk index vector; and comparing the multi-dimensional injury risk index vector with historical health data and a moving target dynamic adjustment threshold value of the subject, and if the threshold value is exceeded, providing real-time auditory, tactile or visual feedback to the subject. According to the method, the influence of movement speed and rhythm change is overcome, dynamic instability in movement can be more sensitively reflected, and personalized risk assessment in a real sense is realized.

The invention discloses a sleep aiding method and system based on brain wave data, and the method comprises the steps: synchronously collecting brain waves, electrocardiosignals, environmental noise and user voice data through a flexible electrode array, constructing a multi-mode causal graph, and dynamically recognizing the type of a noise source (electromagnetic interference or psychological noise); the method comprises the following steps of: generating a target antagonistic intervention signal (such as reverse sound wave counteracting electromagnetic noise and binaural rhythm relieving psychological pressure) according to the target antagonistic intervention signal, generating dynamically adaptive light pulse and tactile vibration parameters in combination with a psychological-physiological collaborative model, and synchronously outputting sound, light and tactile intervention signals through a multi-modal actuator. Through causal reasoning and multi-sensory cooperative regulation and control technologies, accurate inhibition of noise interference and personalized induction of the sleep state are realized, the sleep time is remarkably shortened, the deep sleep duration is prolonged, and the method is particularly suitable for complex noise environments and anxiety-related insomnia scenes.

Disclosed are devices, systems and methods for acquiring, analyzing, and utilizing neurofeedback to promote brain coherence. Neurofeedback is a form of biofeedback that allows an individual to regulate his / her brain activity by providing a visual metaphor of brain function, thereby making it accessible for manipulation. In some embodiments of the present technology, a system includes a brain signal detection device wearable by a subject and a computer device including a display and a brain-computer interface (BCI) configured to monitor brain signals and display visual, auditory, and / or tactile stimuli to the subject according to a neurofeedback threshold-based protocol to deliver brain signal coherence between the left and right hemispheres of a subject's brain.

The invention provides an intelligent vision-guided mechanical arm precise grabbing system. The system comprises a bionic compound eye vision module for collecting multi-angle image information of a target object to generate a dynamic depth map, and a bionic touch sensing module for collecting touch information at the tail end of a mechanical arm to generate a pressure distribution heat map. And the dynamic self-optimization grabbing strategy module is used for generating a grabbing strategy by utilizing the two graphs. And the real-time trajectory planning and obstacle avoidance module optimizes a grabbing path according to the grabbing strategy, the environment obstacle information and the mechanical arm state data, and generates an optimal strategy. The robot can sense the movement of a target object and the change of environmental obstacles in real time, dynamically adjust the force and the posture in the grabbing process, and optimize the obstacle avoidance path in real time. Compared with the prior art, the system has the advantages that the aspects of dynamic obstacle avoidance, target object motion compensation, multi-modal information fusion robustness and the like are remarkably improved, the problem of insufficient adaptability in complex dynamic scenes in the prior art is solved, and high success rate and stability of a precise grabbing task under complex conditions are ensured.

The invention belongs to the field of flexible electronic devices, and particularly relates to a flexible electronic skin system based on a magnetorheological fluid microneedle structure. According to the system, a micro-needle array filled with magnetorheological fluid is constructed on a flexible substrate, and dynamic adjustment of local rigidity and deformation characteristics of the micro-needle array is achieved under the action of an external magnetic field; a sensing electrode array is integrated below or inside the microneedle array; the microneedle array deforms under the action of stress, the distance or contact state change between the microneedle array and the integrated sensing electrode array is converted into capacitance, resistance or impedance signals, multi-point distributed touch or pressure sensing is achieved, and real-time acquisition and output are completed through the sensing electrode array and the signal processing module. The magnetorheological fluid is encapsulated in the hollow microneedle array structure for the first time, an arrayed flexible arrangement form is constructed, dynamic reversible adjustment of the rigidity of the microneedle unit between a liquid state and a solid-like state under the action of an external magnetic field is achieved, and the micro-needle array structure is suitable for flexible wearable equipment and a bionic interaction interface.

The invention discloses an auditory disorder patient rehabilitation training system based on virtual reality. The system comprises a perception fusion module, a basic perception reconstruction module, a language content integration module, a rehabilitation training module and a virtual reality auxiliary module. The invention belongs to the technical field of auditory rehabilitation training, and particularly relates to an auditory disorder patient rehabilitation training system based on virtual reality, which adopts a multi-stage cross-modal attention-enhanced voice audiovisual characterization method, converts sound signals into visual light waves and tactile vibration data through a deep learning model, and displays the visual light waves and tactile vibration data on the basis of the visual light waves and tactile vibration data. High-precision and real-time multi-modal data fusion is realized; basic auditory reconstruction is carried out by adopting an audio frequency generation dynamic adjustment method combined with voiceprint dynamic mapping, personalized sound scene mapping is effectively and dynamically generated, and a patient is helped to reconstruct basic perception ability for sound; a cross-modal contrast learning method is adopted to assist lip language synchronous training, and voice context background prompts are provided in the rehabilitation training process.

The invention belongs to the technical field of robots, and particularly discloses a multi-mode dexterous hand fusing visual tactile sense and palmtop eye and application thereof.The multi-mode dexterous hand comprises a hand body, a visual tactile sensor, a palmtop eye module and a processor, the visual tactile sensor comprises a skeleton, an elastic body, RGB and an NIR camera, the skeleton is of a finger-shaped structure, and the skeleton and the hand body form a hand structure; the elastomer covers the outer side of the framework, and a diffuse reflection coating covers the surface of the elastomer; the RGB camera and the NIR camera are installed in the framework and used for obtaining an RGB image and an NIR image when the first object makes contact with the elastic body; the handheld eye module is installed in the middle of the hand body, comprises a second RGB camera and a TOF camera and is used for taking a second object RGB image and sparse depth; and the processor determines the first object surface three-dimensional point cloud and the second object depth map to realize pose alignment of the two. According to the invention, touch-vision-depth multi-mode perception can be realized, and the precision and robustness of the manipulator are improved.

The present invention relates to the technical field of tire detection, and particularly to a method and system for detecting internal bubble defects of tires based on tactile perception. The method includes fixing and calibrating the spatial coordinates of the tire to be detected, driving a three-axis force sensor to scan row by row along the tire surface by using a multi-axis motion mechanism, collecting the force data in the X, Y, and Z directions under different applied normal forces, and performing noise filtering, zero-point drift correction, and interpolation processing on the data to construct a position-mechanical feature data set. By constructing a multi-force-level mechanical fingerprint map and comparing it with a standard tire mechanical fingerprint library, abnormal low-stiffness or deformation regions are automatically identified, and the bubble size and defect severity are estimated. This method overcomes the disadvantages of high cost, complex operation, and low detection efficiency of traditional X-ray and ultrasonic detection equipment, and has the advantages of high detection accuracy, good real-time performance, and strong adaptability, and is suitable for on-line detection and quality control of internal defects in tire production, retreading, and maintenance.

The invention discloses a tactile finger, a mechanical palm and a robot, and the tactile finger comprises a knuckle base body which is provided with a mounting surface arranged in a curved surface manner; the touch sensing assembly comprises a plurality of electrode attaching units used for sensing the external contact state and collecting pressure data, the electrode attaching units are arranged on the mounting surface at intervals, and each electrode attaching unit comprises a plurality of electrode plates which are communicated with one another; the flexible coating part is used for coating the plurality of electrode pasting and coating units; according to the invention, the plurality of electrode pasting units which are arranged at intervals are arranged on the knuckle base body with the curved mounting surface, and the sensing area is integrally covered by the flexible covering piece, so that compact integration and effective protection of the tactile sensing assembly are realized, the distribution rationality and conduction continuity of the sensing function are ensured, and the sensing effect is improved. The flexibility and adaptability of the whole structure of the tactile finger are also improved, and the tactile finger has good installation adaptability and engineering implementability.

The invention discloses a multi-mode visual tactile sensor and a sensing method thereof. The sensor is composed of a sensing cover and an imaging box body, the sensing cover is fixed on the imaging box body, the sensing cover is provided with a flexible sensing layer, an embedded mark point array combining inert ink and thermochromic ink is arranged in the sensing layer, during tactile sensing, a light source is provided through the imaging box body, and multi-mode sensing data is obtained. Therefore, displacement tracking and temperature sensing are realized at the same time. According to the invention, multi-mode sensing of five modes of a two-dimensional texture pattern, a three-dimensional texture image, a three-dimensional displacement field, a three-dimensional force field and a two-dimensional temperature field can be realized under a single optical frame, real-time co-registration measurement can be realized, and a high-integration-level solution is provided for a robot touch system. The problems that an existing visual tactile sensor cannot sense multi-mode information synchronously, force field reconstruction calculation is complex, and temperature sensing capacity is lacked are solved, and the visual tactile sensor is suitable for application scenes such as robot tactile sensing, man-machine interaction, intelligent artificial limbs and industrial detection.

This invention is a method, device, or system to guide a person concerning how to operate a conventional phone to record more-accurate images of a pathology or injury (e.g. such as a wound or skin lesion) on a body for remote medical evaluation purposes. In an example, machine learning and / or artificial intelligence can be used to guide the person concerning how to move a phone along a virtual arcuate path or shape in space over, across, or around the pathology or injury. In an example, this guidance can be auditory, visual, and / or tactile.

The invention discloses an aircraft skin assembly control method and system, and the method comprises the following steps: segmenting a skin and a frame from an image based on a skin surface image through employing a deep learning segmentation model; registering and splicing the images captured by different spectrum cameras to obtain a complete skin image; based on the complete skin image, using a region expansion method to identify and track a skin surface deformation region; preprocessing the original skin model through a finite element method to obtain a corresponding grid model; estimating stress distribution in the current assembly state based on the force haptic image and a stress transfer model; and based on the deformation area and the stress distribution, generating an assembly path and a pressure adjustment instruction by adopting a self-adaptive control strategy, and controlling the assembly process. Through fusion of a multispectral vision technology and a high-precision force tactile perception technology and combination of a deep learning algorithm and a nonlinear mechanical model, accurate identification of skin surface features and accurate positioning of a deformation area are realized, and the assembly control precision is improved.