Terminal and method for providing vibration feedback on basis of user behavior recognition

Abstract

The present invention relates to a terminal for providing vibration feedback on the basis of user behavior recognition. The terminal possessed by a user comprises: a memory for storing at least one instruction; and a processor, wherein the at least one instruction is executed by the processor so that the terminal: acquires a classification result about a counterpart user behavior type obtained by providing, to a pre-trained behavior type classification model, data generated by detecting the movement of the counterpart user in one or more wearable patches attached to the body of the counterpart user; acquires a vibration pattern corresponding to the acquired classification result on the basis of a mapping table in which a predetermined vibration pattern is defined for respective behavior types; and controls that vibration is generated according to the acquired vibration pattern in the wearable patch attached to the body of the user.

Description

User behavior recognition-based vibration feedback providing terminal and method

[0001] The present invention relates to a terminal and a method for providing vibration feedback based on user behavior recognition, and more specifically, to a technology for controlling a wearable patch to generate vibration based on the user's behavior type.

[0002]

[0003] Existing VR (virtual reality) and AR (augmented reality)-based interaction systems and remote collaboration technologies have primarily focused on conveying information based on the user's visual or auditory stimuli. While these methods can function as basic means of communication, they have limitations in realizing user immersion and realistic interaction.

[0004] In addition, existing wearable devices and sensor-based systems that recognize user movements face many limitations in accurately analyzing and classifying real-time movement data due to the lack of precision in sensors and the performance of data processing algorithms. In particular, in application fields requiring high-precision motion recognition, such as AR / VR environments, remote rehabilitation, and sports training, these technical limitations can lead to reduced immersion and decreased usability.

[0005] In addition, in environments where multiple wearable devices are used simultaneously, latency and synchronization issues in wireless communication between devices frequently occur. This hinders the real-time processing and transmission of motion data, disrupting natural interaction between users and acting as a major factor in reducing system reliability.

[0006] Consequently, existing technologies have limitations such as unidirectional transmission methods biased toward visual and auditory information, motion recognition technology lacking precision and real-time capability, and insufficient communication stability between multiple devices. Due to these issues, intuitive and immediate two-way remote communication based on touch has not been realized.

[0007]

[0008] The problem to be solved according to one embodiment includes providing a vibration feedback provision technology based on user behavior recognition.

[0009] However, the aforementioned problems are not limited to those mentioned above, and other problems not mentioned but intended to be solved will be clearly understood by those skilled in the art to which the present invention pertains from the description below.

[0010]

[0011] A terminal possessed by a user according to a first embodiment includes a memory that stores at least one instruction; and a processor, wherein the at least one instruction is executed by the processor, thereby the terminal obtains a classification result for the behavioral type of the other user obtained by providing data generated by detecting the movement of the other user from one or more wearable patches attached to the other user's body to a pre-learned behavioral type classification model, obtains a vibration pattern corresponding to the obtained classification result based on a mapping table in which a predetermined vibration pattern is defined for each of the plurality of behavioral types, and controls the wearable patches attached to the user's body to generate vibration according to the obtained vibration pattern.

[0012] In addition, the behavior type classification model can output a classification result for the behavior type of the counterpart by additionally considering information regarding the number and attachment location of wearable patches attached to the counterpart user's body.

[0013] In addition, the information additionally considered above may be obtained from pre-specified attachment location information for each wearable patch attached to the body of the counterpart user.

[0014] In addition, the information additionally considered above may be obtained from the number of attached wearable patches and attachment location information entered by the counterpart user through their terminal.

[0015] Additionally, as the above at least one instruction is executed by the processor, information regarding the number and attachment location of wearable patches attached to the user's body is additionally obtained, and the mapping table may be generated based on the additionally obtained information.

[0016] In addition, a mapping table generated based on the additionally acquired information can be displayed to the user.

[0017] Additionally, the data generated by detecting the movement of the aforementioned user includes linear acceleration data in three axis directions, and the behavior type classification model can be pre-trained to generate a time-frequency spectrum by applying a continuous wavelet transform (CWT) to the linear acceleration data when the linear acceleration data is input, and to output a classification result for the behavior type based on a feature vector extracted from the time-frequency spectrum.

[0018] In addition, when at least one of the above commands is executed by the processor, if a wearable patch is not attached to a part of the user's body corresponding to the acquired classification result, it is possible to control vibration to occur through a wearable patch attached to a selected part according to a predefined priority.

[0019] In addition, if a wearable patch is not attached to the area selected according to the above priority, the acquired classification result can be provided in the form of at least one of a text message and a voice message.

[0020] A method for providing vibration feedback of a terminal possessed by a user according to a second embodiment may include: a step of obtaining a classification result for a behavioral type of a relative user obtained by providing data generated by detecting the movement of the relative user from one or more wearable patches attached to the relative user's body to a pre-learned behavioral type classification model; a step of obtaining a vibration pattern corresponding to the obtained classification result based on a mapping table in which a predetermined vibration pattern is defined for each of a plurality of behavioral types; and a step of controlling the wearable patches attached to the user's body to generate vibration according to the obtained vibration pattern.

[0021] A computer-readable recording medium storing a computer program according to a third embodiment, wherein the computer program may include instructions for the processor to perform a method comprising: a step of obtaining a classification result for a behavioral type of a counterpart user obtained by providing data generated by detecting the movement of the counterpart user from one or more wearable patches attached to the body of the counterpart user of a terminal held by the user to a pre-learned behavioral type classification model when executed by a processor; a step of obtaining a vibration pattern corresponding to the obtained classification result based on a mapping table in which a predetermined vibration pattern is defined for each of a plurality of behavioral types; and a step of controlling the wearable patches attached to the body of the user to generate vibration according to the obtained vibration pattern.

[0022]

[0023] According to one embodiment, a new communication method can be provided through a skin-attached wearable patch and a vibration actuator that intuitively conveys and allows the user's movements and intentions to be felt, and immersive tactile-based interaction can be realized.

[0024] Furthermore, by utilizing IMU sensor-based data and machine learning classification algorithms, user behavior patterns are precisely analyzed in real time, significantly improving data processing efficiency and responsiveness in various application fields such as VR / AR, rehabilitation, and remote communication.

[0025] In addition, high-performance BLE-based wireless communication ensures a stable connection without time delay, and guarantees a seamless communication environment even when multiple devices are used simultaneously.

[0026] In addition, since it automatically detects user movements and provides feedback based on them, an intuitive interaction environment close to everyday life can be realized.

[0027] The effects obtainable from the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs from the description below.

[0028]

[0029] FIG. 1 is an overall conceptual diagram of a vibration feedback providing system according to one embodiment.

[0030] FIG. 2 is an exemplary configuration diagram of the wearable patch and terminal of FIG. 1.

[0031] FIG. 3 is a flowchart illustrating the process of a terminal of a vibration feedback providing system providing vibration feedback according to one embodiment.

[0032] Figure 4 is a diagram showing the subdivided functional modules of the wearable patch, terminal, and server of Figure 1.

[0033] Figure 5 is a diagram showing the structure of a behavior type classification model according to one embodiment.

[0034] FIG. 6 is an exploded view illustrating the internal components of a wearable patch according to one embodiment.

[0035]

[0036] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but can be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims.

[0037] In describing the embodiments of the present invention, specific descriptions of known functions or configurations will be omitted if it is determined that such detailed descriptions could unnecessarily obscure the essence of the invention. Furthermore, the terms described below are defined in consideration of their functions in the embodiments of the present invention, and these definitions may vary depending on the intentions or practices of the user or operator. Therefore, such definitions should be based on the content throughout this specification.

[0038] Below, embodiments of the present invention are described in detail with reference to the drawings so that those skilled in the art can easily implement the present invention.

[0039] FIG. 1 is an overall conceptual diagram of a vibration feedback providing system according to one embodiment.

[0040] The illustrated vibration feedback providing system includes a wearable patch (110) attached to the user's body, the user's terminal (120), a server (130), a wearable patch (150) attached to the other user's body, and the other user's terminal (140). Here, the terminal (120, 140) may be, for example, a smartphone, and may also be various types of electronic devices such as a tablet PC, smartwatch, smart glasses, laptop computer, AR / VR HMD (head mounted display), smart band, portable game console, wearable remote control, dedicated portable terminal, etc.

[0041] A wearable patch (110, 150) is attached to various parts of the user's body (e.g., arms, legs, waist, etc.) to detect the user's movements and acquire a vibration pattern corresponding to the user's actions, and generates vibrations according to the acquired vibration pattern.

[0042] The terminal (120, 140) detects the user's movement from the wearable patch (110, 150), collects the generated data, and transmits it to the server (130). It also receives information from the user regarding the number and attachment locations of the wearable patches attached to the user's body and transmits this information to the server (130). Additionally, the terminal (120, 140) generates a mapping table in which vibration patterns are defined for each type of behavior and provides it to the user. Here, the mapping table may be provided in a form displayed on the screen of the terminal (120, 140), but is not limited thereto.

[0043] The server (130) detects the movement of a user from a wearable patch (110, 150) received from a terminal (120, 140) and provides the generated data to a pre-trained behavioral type classification model to obtain a classification result for the user's behavioral type, and transmits the obtained classification result to the terminal (120, 140).

[0044] Meanwhile, the above behavior type classification model is not limited to being operated on the server (130) and may be embedded within the terminal (120, 140) and executed directly in a local environment. In this case, the terminal (120, 140) receives data generated by detecting the user's movement from the wearable patch (110, 150), processes it internally, classifies the behavior type through the behavior type classification model to determine a corresponding vibration pattern, and then provides the determined vibration pattern to the wearable patch (110, 150) via a control signal.

[0045] Through this configuration, the vibration feedback providing system according to one embodiment can independently provide behavior recognition and vibration feedback functions even in an environment where the network connection is unstable or access to the server (130) is restricted.

[0046] FIG. 2 is an exemplary configuration diagram of the wearable patch and terminal of FIG. 1.

[0047] The wearable patch (110, 150) and terminal (120, 140) according to the embodiment include a communication unit (10), a memory (20), and a processor (30). However, the configuration diagram shown in FIG. 2 is merely exemplary, and the concept of the present invention is not limited to the configuration diagram shown in FIG. 2. For example, the wearable patch (110, 150) and terminal (120, 140) may include at least one configuration not shown in FIG. 2 or may not include at least one of the configurations shown in FIG. 2.

[0048] The communication unit (10) can be implemented by various types of wired or wireless communication modules.

[0049] For example, the terminal (120, 140) can perform BLE (Bluetooth Low Energy) based wireless communication with the wearable patch (110, 150) through the communication unit (10) and perform network-based communication with the server (130). At this time, the communication unit (10) can provide bidirectional communication functions for receiving data generated by detecting the user's movement in the wearable patch (110, 150), transmitting and receiving classification results for behavior types, and transmitting control signals related to vibration patterns.

[0050] For example, the wearable patch (110, 150) can perform BLE-based wireless communication with the terminal (120, 140) through the communication unit (10). Additionally, the wearable patch (110, 150) can transmit data generated by detecting the user's movement to the terminal (120, 140) through the communication unit (10), and receive control signals related to vibration patterns from the terminal (120, 140). At this time, the communication unit (10) can provide bidirectional communication functions for transmitting data generated by detecting the user's movement, signal processing for synchronization with other wearable patches attached to the same user, and receiving control signals related to vibration patterns.

[0051] Memory (20) can be implemented by a medium that stores information. Such a medium may be at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (Random Access Memory, RAM), SRAM (Static Random Access Memory), ROM (Read-Only Memory, ROM), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and optical disk, but is not limited thereto.

[0052] Various types of information may be stored in such memory (20). For example, the terminal (120, 140) may store information regarding the number and attachment location of wearable patches attached to the user's body, which are input from the user through the memory (20), a mapping table in which vibration patterns by behavior type are defined, and other received or generated data.

[0053] Meanwhile, the processor (30) can perform technical features according to embodiments of the present disclosure to be described later by executing at least one instruction stored in memory (20). In one embodiment, the processor (30) may be composed of at least one core and may include a processor for data analysis and / or processing, such as a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), or a tensor processing unit (TPU) of a terminal (120, 140) or a wearable patch (110, 150).

[0054] In the following, the process of a terminal (120, 140) providing vibration feedback based on user behavior recognition is explained by executing at least one instruction stored in memory (120) by the processor (130) through FIGS. 3 to 5.

[0055] FIG. 3 is a flowchart illustrating the process of a terminal providing vibration feedback according to one embodiment, FIG. 4 is a diagram illustrating the subdivided functional modules of the wearable patch, terminal, and server of FIG. 1, and FIG. 5 is a diagram illustrating the structure of a behavior type classification model according to one embodiment. For reference, in the embodiment to be described below, only the modules directly related to the invention among the subdivided functional modules shown in FIG. 4 will be mentioned.

[0056] A terminal possessed by a user of a vibration feedback providing system according to one embodiment obtains a classification result of the other party's behavior type through a pre-learned behavior type classification model (S301). The behavior type refers to a specific action classified according to the user's body movement, and may include various action types such as walking, running, jumping, sitting, standing, arm raise, hand wave, hug, turn, squat, hook, punch, pick, and jump pack.

[0057] The above behavior type classification model is a model that is pre-trained to classify corresponding behavior types by receiving data generated by detecting the user's movements from one or more wearable patches attached to the other person's body, and thus the classification result for the above behavior type is obtained by providing the data generated by detecting the user's movements to the above behavior type classification model.

[0058] Here, the behavior type classification model can be trained using a supervised learning method, and as the data for training, various types of data generated by detecting the movements of multiple users can be utilized as input, and as the correct answer, information indicating what type of behavior it is for each input data can be utilized.

[0059] In addition, the above behavior type classification model can output a more accurate behavior type classification result by additionally considering information regarding the number and attachment location of wearable patches attached to the other person's body.

[0060] The additional information considered above may be obtained from pre-specified attachment location information for each wearable patch attached to the body of the other party, or from the number of wearable patches and attachment location information directly entered by the other party through their terminal.

[0061] For example, if a total of three wearable patches are attached to the opponent's body and are pre-registered to be located on the left arm, right arm, and waist, respectively, the behavioral type classification model can recognize precise behavioral types, such as "a punching motion originating from the left arm" or "a rotational motion centered on the waist," by analyzing the user's movement data detected at each location along with the attachment location information, going beyond the level of simply "there is movement."

[0062] Meanwhile, the data generated by detecting user movement provided to the behavior type classification model includes linear acceleration data for three axis directions (X, Y, Z) measured through an IMU sensor included in the sensing and feedback module among the functional modules of the wearable patch (110, 150) shown in FIG. 4. That is, the wearable patch (110, 150) measures linear acceleration for three axis directions through the IMU sensor and transmits it to the terminal (120, 140), and the terminal (120, 140) monitors the linear acceleration in real time and transmits the linear acceleration data to the server if the linear acceleration value exceeds a predetermined threshold value.

[0063] When linear acceleration data is input into the behavior type classification model through this process, the above

[0064] The behavior type classification model applies a continuous wavelet transform (CWT) to input linear acceleration data to generate a time-frequency spectrum representing the energy distribution of time and frequency. In one embodiment, a Morse wavelet function is used to precisely extract motion patterns in the low-frequency region, and the result is visualized as two-dimensional image data called a scalogram. Figure 5 illustrates an example in which X-axis, Y-axis, and Z-axis acceleration data are individually converted into scalogram images.

[0065] Subsequently, the behavior type classification model inputs three scalar images into a convolutional neural network (CNN) structure based on the AlexNet architecture, and extracts feature maps by applying convolution layers and pooling layers to each image. Then, the extracted feature maps are converted into a single feature vector by applying flatten layers and dense layers, and finally, the output of the dense layer classifies which behavior type the user's movement corresponds to. Figure 5 illustrates an example where the result classified as 'punch' among the behavior types is selected as the final output value.

[0066] A terminal that has obtained a classification result for the behavior type of the counterparty through S301 obtains a vibration pattern corresponding to the obtained classification result based on a mapping table in which a predetermined vibration pattern is defined for each of the multiple behavior types (S303). For example, if a behavior type called 'punch' is classified, a vibration pattern such as "two short vibrations at 1.5-second intervals" defined in the mapping table for that behavior type may be obtained.

[0067] At this time, the terminal may additionally obtain information regarding the number and location of wearable patches attached to the user's body, and in this case, a mapping table may be generated based on the additionally obtained information. Accordingly, the mapping table may be composed of a table that defines the location generating vibration and the vibration pattern together by reflecting the number and location of the user's wearable patches, as well as a one-to-one correspondence between the behavior type and the vibration pattern.

[0068] Meanwhile, the mapping table generated in this way can be displayed to the user through the data visualizing module among the function modules of the terminal (120, 140) shown in FIG. 4.

[0069] The terminal, having obtained a vibration pattern through S303, controls the wearable patch attached to the user's body to generate vibration according to the obtained vibration pattern (S305). At this time, the vibration is generated through a vibration actuator (vibro-haptic actuator) included in the sensing and feedback module among the functional modules of the wearable patch (110, 150) shown in FIG. 4, and the generated vibration is output through the wearable patch attached to the body part directly corresponding to the classified action type. For example, if the opponent's action is classified as 'punch' and the corresponding action type is mapped to "vibration output to left arm," the terminal controls the wearable patch attached to the user's left arm to output the corresponding vibration pattern.

[0070] However, if a wearable patch is not attached to the body part corresponding to the classification result, an alternative part is determined according to a priority predefined by the terminal. For example, a priority such as "left arm → right arm → shoulder → waist → skip vibration" may be predefined according to user settings or system built-in rules, and a part to output vibration may be selected according to that priority.

[0071] For example, if the classified behavior type is 'left arm rotation' or 'left arm punch', or if a wearable patch is not attached to the user's left arm, the terminal selects a patch attached to the right arm or shoulder according to priority and controls the patch to generate vibration.

[0072] In addition, if a wearable patch is not attached to the area selected according to priority, the terminal may provide a text message or voice message to the user as an alternative feedback means. These messages may be provided, for example, as "Left arm usage detected. No patch currently attached to that area," or "Punch motion detected. No vibration output possible," and serve as an auxiliary notification means to inform the user of the results of the behavior type classification.

[0073] FIG. 6 is an exploded view illustrating the internal components of a wearable patch according to one embodiment.

[0074] A wearable patch according to one embodiment is in a form that can be directly attached to a user's skin and includes a vibration actuator, an IMU sensor, a microcontroller, a passive element, and a copper electrode, and all of these components are wrapped in biocompatible silicone.

[0075] The vibration actuator and IMU sensor above perform the same function as the vibration actuator and IMU sensor shown in FIG. 4, and the microcontroller performs the same function as the communication module (wireless module) shown in FIG. 4.

[0076] The above passive components include various electronic components such as resistors, capacitors, and inductors included for purposes such as filtering, power stabilization, and impedance matching, and the copper electrodes form a circuit pattern for electrical connection between these components.

[0077] The wearable patch configured in this way is made of biocompatible silicone that causes minimal irritation even when attached to the user's skin for a long time, and the biocompatible silicone can protect the components and increase adhesion to the skin.

[0078] Meanwhile, the method according to the various embodiments described above may be implemented in the form of a computer program stored on a computer-readable recording medium programmed to perform each step of the method, and may also be implemented in the form of a computer-readable recording medium storing a computer program programmed to perform each step of the method.

[0079] The above description is merely an illustrative explanation of the technical concept of the present invention, and those skilled in the art to which the present invention pertains will be able to make various modifications and variations within the scope of the essential quality of the present invention. Accordingly, the embodiments disclosed in the present invention are intended to explain, not limit, the technical concept of the present invention, and the scope of the technical concept of the present invention is not limited by such embodiments. The scope of protection of the present invention shall be interpreted by the claims below, and all technical concepts within the equivalent scope shall be interpreted as being included within the scope of rights of the present invention.

Claims

1. A vibration feedback providing terminal based on user behavior recognition possessed by a user, Memory for storing at least one instruction; and Includes a processor, By executing the above at least one instruction by the processor, The above terminal is, A classification result for the behavioral type of the other user is obtained by providing data generated by detecting the movement of the other user from one or more wearable patches attached to the other user's body to a pre-trained behavioral type classification model, and Based on a mapping table in which a predetermined vibration pattern is defined for each of a plurality of behavior types, a vibration pattern corresponding to the acquired classification result is obtained, and To the vibration pattern obtained from the wearable patch attached to the body of the user Controlling to generate vibrations accordingly A vibration feedback providing terminal based on user behavior recognition.

2. In Paragraph 1, The above behavioral type classification model is, Outputting a classification result for the behavioral type of the aforementioned user by additionally considering information regarding the number and attachment location of wearable patches attached to the body of the aforementioned user. A vibration feedback providing terminal based on user behavior recognition.

3. In Paragraph 2, The additional information considered above is, That which is obtained from pre-specified attachment location information for each wearable patch attached to the body of the aforementioned counterpart user A vibration feedback providing terminal based on user behavior recognition.

4. In Paragraph 2, The additional information considered above is, The information obtained from the number of attached wearable patches and attachment location information entered by the aforementioned counterpart user through their terminal A vibration feedback providing terminal based on user behavior recognition.

5. In Paragraph 1, By executing the above at least one instruction by the processor, Information regarding the number and attachment locations of wearable patches attached to the body of the above-mentioned user is additionally obtained, and The above mapping table is, It is generated based on the additionally acquired information mentioned above. A vibration feedback providing terminal based on user behavior recognition.

6. In Paragraph 5, A mapping table generated based on the additionally acquired information is displayed to the user. A vibration feedback providing terminal based on user behavior recognition.

7. In Paragraph 1, The data generated by detecting the movement of the aforementioned counterpart user includes linear acceleration data for three axis directions, and The above behavioral type classification model is, When the above linear acceleration data is input, a continuous wavelet transform (CWT) is applied to the above linear acceleration data to generate a time-frequency spectrum, and a pre-trained model is provided to output a classification result for the above behavior type based on a feature vector extracted from the above time-frequency spectrum. A vibration feedback providing terminal based on user behavior recognition.

8. In Paragraph 1, By executing the above at least one instruction by the processor, If a wearable patch is not attached to a part of the user's body corresponding to the acquired classification result, control is made to generate vibration through a wearable patch attached to a selected part according to a predefined priority. A vibration feedback providing terminal based on user behavior recognition.

9. In Paragraph 8, If a wearable patch is not attached to the area selected according to the above priority, the acquired classification result is provided in the form of at least one of a text message and a voice message. A vibration feedback providing terminal based on user behavior recognition.

10. A method for providing vibration feedback of a terminal possessed by a user, A step of obtaining a classification result for the behavioral type of the counterpart user obtained by providing data generated by detecting the movement of the counterpart user from one or more wearable patches attached to the counterpart user's body to a pre-trained behavioral type classification model; A step of obtaining a vibration pattern corresponding to the obtained classification result based on a mapping table in which a predetermined vibration pattern is defined for each of a plurality of behavior types; and A step of controlling vibration to occur in a wearable patch attached to the body of the user according to the acquired vibration pattern; Method for providing vibration feedback of a terminal.

11. A non-transient computer-readable recording medium storing a computer program, When the above computer program is executed by a processor, A step of obtaining a classification result for the behavioral type of the counterpart user obtained by providing data generated by detecting the movement of the counterpart user from one or more wearable patches attached to the body of the counterpart user of a terminal possessed by the user to a pre-trained behavioral type classification model; A step of obtaining a vibration pattern corresponding to the obtained classification result based on a mapping table in which a predetermined vibration pattern is defined for each of a plurality of behavior types; and A non-transient computer-readable recording medium comprising instructions for the processor to perform a method including the step of controlling vibration to occur according to the acquired vibration pattern in a wearable patch attached to the body of the user.

Images