Method for SNN-based control of robotic arm imitating arm motion by using EMG and dvs

Abstract

A neuromorphic device for imitating and reproducing a person's hand or arm motion according to the present invention may comprise: a communication unit for receiving information on an electromyography (EMG) signal related to a motion of a person's hand or arm measured by an EMG sensor, and receiving information on a movement of the person's hand or arm detected by a dynamic vision sensor (DVS) camera; and a processor for converting the received information on the EMG signal into a spike signal, applying, to a predetermined trained spiking neural network (SNN) model, the converted spike signal and the information on the movement of the person's hand or arm received from the DVS camera, and outputting a class corresponding to the motion of the person's hand or arm among classes defined in advance on the basis of movement and position.

Description

SNN-based arm motion mimicking robot arm control method using EMG and DVS

[0001] The present invention relates to an SNN-based arm motion mimicking robot arm control method using EMG and DVS and a neuromorphic device for the same.

[0002] This research was conducted as a result of the University ICT Research Center Development Support Project of the Ministry of Science and ICT and the Korea Institute of Information and Communications Technology Planning and Evaluation (IITP). (IITP-2024-RS-2022-00156225)

[0003] An exoskeleton robot system utilizing electromyography signals has the effect of providing a robot arm mechanism system capable of easily lifting heavy objects by extracting electromyography signals measured from a person's arm and hand, filtering and processing these signals to drive the actuator.

[0004] According to the robot motion control device and method using an electromyography sensor and an accelerometer, there is an advantage in that a user can easily remotely control the operation of a robot by utilizing signals from an electromyography sensor and an accelerometer mounted on the human body.

[0005] As such, conventional methods of controlling robot motion using electromyography and accelerometer sensors could not precisely and accurately mimic hand and arm movements, and the inability to control robot motion quickly resulted in poor real-time responsiveness. Furthermore, the high power consumption required for robot control limited the development of medical robots.

[0006] Accordingly, the present invention proposes a solution to overcome the limitations of existing medical robot motion control.

[0007] The technical problem to be solved by the present invention is to provide a neuromorphic device for mimicking and reproducing human hand or arm movements.

[0008] Another technical objective of the present invention is to provide a method for a neuromorphic device to mimic and reproduce human hand or arm movements.

[0009] Another technical objective of the present invention is to provide a neuromorphic-based system for mimicking and reproducing human hand or arm movements.

[0010] Another technical objective of the present invention is to provide a computer-readable recording medium that records a program for executing on a computer a method for a neuromorphic device to mimic and reproduce human hand or arm movements.

[0011] The technical problems to be solved by the present invention are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which the present invention belongs from the description below.

[0012] A neuromorphic device for mimicking and reproducing a human hand or arm motion according to the present invention, for achieving the above technical objectives, comprises: a communication unit that receives information regarding an electromyography signal related to the motion of a human hand or arm measured by an electromyography (EMG) sensor and receives information regarding the movement of the human hand or arm detected by a dynamic vision sensor (DVS) camera; and a processor that converts the information regarding the received electromyography signal into a spike signal, applies the converted spike signal and the information regarding the movement of the human hand or arm received from the DVS camera to a predetermined learned spiking neural network (SNN) model, and outputs a class corresponding to the motion of the human hand or arm among classes defined in advance based on movement and position.

[0013] The above communication unit is characterized by transmitting the above-mentioned output class to a robot to reproduce the above-mentioned human hand or arm movements by mimicking them.

[0014] The processor is characterized by acquiring electromyography data of the person's hand or arm, extracting features from the electromyography data of the person's hand or arm using an adaptive filter, and then performing a spike transformation. The processor is characterized by extracting features using the adaptive filter and then performing a spike transformation using delta-sigma modulation.

[0015] A method for a neuromorphic device to mimic and reproduce a human hand or arm movement to achieve other technical objectives described above may include: receiving information regarding an electromyogram signal related to a human hand or arm movement measured from an EMG sensor; receiving information regarding the movement of the human hand or arm detected by a Dynamic Vision Sensor (DVS) camera; converting the information regarding the received electromyogram signal into a spike signal; inputting the converted spike signal and the information regarding the human hand or arm movement received from the DVS camera into a predetermined learned Spiking Neural Network (SNN) model; and outputting a class corresponding to the movement of the human hand or arm among classes defined in advance based on movement and position.

[0016] The above method further includes the step of transmitting the output class to a robot to reproduce the human hand or arm movements by mimicking them.

[0017] The step of converting into a spike signal comprises: a step of acquiring electromyography data of the person's hand or arm; and a step of extracting features from the electromyography data of the person's hand or arm using an adaptive filter and then performing spike conversion. The step of performing spike conversion is characterized by performing spike conversion using delta-sigma modulation.

[0018] A neuromorphic-based system for mimicking and reproducing human hand or arm movements to achieve another technical objective described above comprises: an EMG sensor for measuring electromyogram signals related to the movements of a human hand or arm; a Dynamic Vision Sensor (DVS) camera for detecting the movements of the human hand or arm; and a processor for converting information regarding the electromyogram signals measured by the EMG sensor into spike signals, and applying the converted spike signals and information regarding the human hand or arm movements output by the DVS camera to a predetermined learned Spiking Neural Network (SNN) model to output a class corresponding to the movements of the human hand or arm among classes defined in advance based on movement and position.

[0019] By classifying data obtained through the EMG sensor and DVS camera according to the present invention or outputting a class using an SNN model, it has become possible to control the robot quickly and precisely with low power consumption.

[0020] The effects obtainable from the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description below.

[0021] The accompanying drawings, which are included as part of the detailed description to aid in understanding the present invention, provide embodiments of the present invention and explain the technical concept of the present invention together with the detailed description.

[0022] Figure 1 is a diagram illustrating the layer structure of an artificial neural network.

[0023] Figure 2 is a diagram illustrating an example of a deep neural network.

[0024] Figure 3 is a diagram illustrating the spike encoding method for input data when training a Spiking Neural Network (SNN) algorithm model.

[0025] Figure 4 is a diagram illustrating the correlation between input data and output data during SNN model training according to the present invention.

[0026] FIG. 5 is a diagram illustrating a block diagram for explaining the function and configuration of a neuromorphic device (500) according to the present invention.

[0027] FIG. 6 is a diagram illustrating a system for controlling a robot hand / arm by mimicking SNN-based hand / arm movements using an EMG sensor (100) and a DVS camera (200) according to the present invention.

[0028] FIG. 7 is a diagram illustrating the process of a neuromorphic device (500) according to the present invention converting or encoding ECG data into a spike signal.

[0029] FIG. 8 is a diagram illustrating the processing steps performed in the DVS camera (200).

[0030] Referring to Figure 9, the SNN model may be a model trained through unsupervised learning, and about 6 classes were predefined.

[0031] FIG. 10 is a drawing comparing the result of implementing the neuromorphic device (500) according to the present invention to mimic and reproduce the movements of a real person's hand and arm with the prior art.

[0032] Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings. The detailed description disclosed below, together with the accompanying drawings, is intended to describe exemplary embodiments of the present invention and is not intended to represent the only embodiment in which the present invention may be practiced. The following detailed description includes specific details to provide a complete understanding of the present invention. However, those skilled in the art will know that the present invention may be practiced without such specific details.

[0033] In some cases, to avoid obscuring the concept of the present invention, known structures and devices may be omitted or illustrated in the form of block diagrams focusing on the core functions of each structure and device. Additionally, throughout this specification, the same components are described using the same reference numerals.

[0034] The present invention is capable of various modifications and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the invention to specific embodiments, and it should be understood that the invention includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the invention.

[0035] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0036] Terms such as "first," "second," etc., may be used to describe various components, but said components should not be limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.

[0037] The terms used herein are merely for describing specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “comprising” or “having” are intended to indicate the presence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0038] Furthermore, the components of the embodiments described with reference to each drawing are not limited to the respective embodiments and may be implemented to be included in other embodiments within the scope of maintaining the technical spirit of the present invention. It is also obvious that multiple embodiments may be re-implemented as a single embodiment that integrates multiple embodiments, even if a separate description is omitted.

[0039] Additionally, terms such as “…part,” “…unit,” “…module,” and “…device” described in the specification refer to a unit that processes at least one function or operation, and this may be implemented in hardware, software, or a combination of hardware and software.

[0040] Before describing the present invention, we will explain artificial intelligence (AI), machine learning, and deep learning. The easiest way to understand the relationship between these three concepts is to visualize three concentric circles. Artificial intelligence is the largest circle, followed by machine learning, and deep learning, which is leading the current AI boom, can be considered the smallest circle.

[0041] The concept of artificial intelligence first emerged at the Dartmouth Conference hosted by Professor John McCarthy at Dartmouth College in the United States in 1956, and it has been growing explosively in recent years. This growth has been further accelerated, particularly since 2015, by the introduction of GPUs that provide rapid and powerful parallel processing capabilities. The advent of the Big Data era, characterized by explosively increasing storage capacity and a flood of data across all domains—including images, text, and mapping data—has also had a significant impact on this growth trend.

[0042] Artificial Intelligence - Realizing human intelligence in machines

[0043] In 1956, the pioneers of artificial intelligence dreamed of ultimately creating complex computers with characteristics similar to human intelligence. While artificial intelligence that thinks like a human, possessing human senses and thinking abilities, is called 'General AI,' the artificial intelligence achievable at the current level of technological development falls under the concept of 'Narrow AI.' Narrow AI is characterized by its ability to perform specific tasks with capabilities exceeding those of humans, such as image classification services on social media or facial recognition functions.

[0044] Machine Learning - A Specific Approach to Implementing Artificial Intelligence

[0045] Machine learning serves the role of automatically filtering spam from your inbox. Meanwhile, machine learning fundamentally uses algorithms to analyze data, learns through analysis, and performs judgments or predictions based on what it has learned. Therefore, its ultimate goal is not to directly code specific guidelines for decision criteria into the software, but rather to 'train' the computer itself through massive amounts of data and algorithms to learn how to perform tasks. Machine learning originated from concepts directly proposed by early artificial intelligence researchers, and its algorithmic methods include decision tree learning, inductive logic programming, clustering, reinforcement learning, and Bayesian networks. However, none of these have achieved general AI, which can be considered the ultimate goal, and it is true that early machine learning approaches often struggled to complete even narrow AI.

[0046] Currently, machine learning is achieving significant results in fields such as computer vision, but it has encountered a limitation in that a certain amount of coding work is involved throughout the entire process of implementing artificial intelligence, even without specific guidelines. For instance, when recognizing an image of a stop sign based on a machine learning system, the developer must directly code boundary detection filters that programmatically identify the start and end points of an object, shape detection systems that verify the surface of an object, and classifiers that recognize characters such as 'STO-P'. In this way, machine learning operates by recognizing images from 'coded' classifiers and 'learning' stop signs through algorithms.

[0047] While machine learning achieves sufficient performance for commercialization in image recognition, the accuracy can drop in specific situations where signs are obscured by fog or trees. The reason computer vision and image recognition have not yet reached human levels until recently is due to these recognition rate issues and frequent errors.

[0048] Deep Learning - A technology that enables complete machine learning

[0049] The biological characteristics of the human brain, particularly the connection structure of neurons, inspired artificial neural networks, another algorithm created by early machine learning researchers. However, unlike the brain, where any physically adjacent neurons can be interconnected, artificial neural networks have fixed layer connections and data propagation directions.

[0050] For example, when an image is cut into numerous tiles and input into the first layer of a neural network, the neurons repeat the process of passing data to the next layer until a final output is generated at the last layer. Each neuron is assigned a weight representing the accuracy of the input based on the task performed, and the final output is determined by summing all the weights. In the case of a stop sign, the image's characteristics—such as its octagonal shape, red color, text, size, and movement—are finely cut and 'inspected' by the neurons, and the neural network's task is to identify whether it is a stop sign. Here, a 'probability vector' is utilized to predict the result based on weights derived from sufficient data.

[0051] Deep learning is a form of artificial intelligence that has evolved from artificial neural networks, utilizing information input and output layers similar to the neurons in the brain to learn data. However, because even basic neural networks require a massive amount of computation, the commercialization of deep learning faced obstacles from the beginning. Nevertheless, researchers continued their work and succeeded in parallelizing algorithms that prove the concept of deep learning based on supercomputers. Furthermore, the emergence of GPUs, which are optimized for parallel processing, dramatically accelerated the computational speed of neural networks, leading to the advent of true deep learning-based artificial intelligence.

[0052] Neural networks are highly likely to produce numerous incorrect answers during the 'learning' process. Returning to the example of the stop sign, to precisely adjust the weights of neuron inputs to always produce the correct answer regardless of weather conditions or day-night cycles, one might need to learn from hundreds, thousands, or perhaps even millions of images. Only when this level of accuracy is reached can the neural network be considered to have properly learned the stop sign. In 2012, Google and Stanford University Professor Andrew Ng implemented a 'Deep Neural Network' consisting of over 1 billion neural networks using 16,000 computers. Through this, they extracted and analyzed 10 million images from YouTube and succeeded in having the computer classify photos of people and cats. They enabled the computer to independently learn the process of recognizing and judging the shape and appearance of cats appearing in the videos.

[0053] The image recognition capabilities of systems trained with deep learning have already surpassed those of humans. Furthermore, the scope of deep learning extends to areas such as identifying cancer cells in the blood and tumors in MRI scans. Google's AlphaGo learned the fundamentals of Go and further strengthened its neural network through the process of repeatedly playing matches against AIs similar to itself. The emergence of deep learning has enhanced the practicality of machine learning and expanded the scope of artificial intelligence. Deep learning subdivides tasks in every way possible that can be supported by computer systems. Deep learning-based technologies, such as driverless cars, improved preventive medicine, and more accurate movie recommendations, are already being used in our daily lives or are on the verge of practical application. Deep learning is regarded as both the present and the future of artificial intelligence, possessing the potential to realize the general AI that once appeared in science fiction.

[0054] Below, we will take a closer look at deep learning.

[0055] Deep learning is a type of artificial neural network (ANN) based on human neural network theory. It is a set of machine learning models or algorithms that refer to a deep neural network (DNN) composed of a layer structure and having one or more hidden layers (hereinafter referred to as intermediate layers) between the input layer and the output layer. Simply put, deep learning can be described as an artificial neural network with deep layers.

[0056] The human brain is estimated to be composed of 25 billion nerve cells. The brain consists of nerve cells, and each nerve cell (neuron) refers to a single nerve cell that forms a neural network. A nerve cell contains a cell body, a single axon (or nurite) which is a projection of the cell body, and usually several dendrites (or protoplasmic processes). Information exchange between these nerve cells is transmitted through junctions between nerve cells called synapses. While a single nerve cell appears very simple when viewed in isolation, when these nerve cells come together, they are capable of possessing human intelligence. The dendrites are the part that receives signals sent by other nerve cells (Input), while the axon is the long extension from the cell body that transmits signals to other nerve cells (Output). There is a connection called a synapse that links the axon and dendrite, which transmit signals between nerve cells; however, the signal is not transmitted unconditionally, but is only transmitted when the signal strength exceeds a certain value (threshold). In other words, not only is the connection strength different for each synapse, but it also determines whether or not to transmit a signal.

[0057] Artificial neural networks (ANNs), a field of artificial intelligence, are mathematical models modeled by mimicking the structure of the biological (typically human) brain (neural networks). In other words, artificial neural networks are implemented by imitating the information processing and transmission processes of these biological neurons. As they are implemented similarly to how the human brain solves problems, neural networks possess excellent parallelism because each neuron operates independently. Furthermore, since information is distributed across numerous connections, problems in a few neurons do not significantly affect the entire network; consequently, they are resilient to a certain level of error and possess the ability to learn from a given environment.

[0058] Deep neural networks can be viewed as descendants of artificial neural networks. They are the latest version of artificial neural networks, having overcome existing limitations and achieved success in areas where numerous artificial intelligence technologies had previously failed. When examining the modeling of artificial neural networks that mimic biological neural networks, biological neurons are modeled as nodes in terms of processing units, and synapses are modeled as weights in terms of connections, as shown in Table 1 below.

[0059] Biological Neural Network Artificial Neural Network Cell Body Node Dendrite Input Axon Output Synapse Weight

[0060] Figure 1 is a diagram illustrating the layer structure of an artificial neural network.

[0061] Just as human biological neurons perform meaningful tasks by connecting multiple cells rather than just one, artificial neural networks connect individual neurons to one another through synapses, creating multiple interconnected layers where the connection strength between layers can be updated using weights. In this way, they are utilized in fields for learning and cognition through their multi-layered structure and connection strengths.

[0062] Each node is connected by weighted links, and the entire model learns by repeatedly adjusting these weights. Weights represent the importance of each node as a fundamental means for long-term memory. Simply put, an artificial neural network trains the entire model by initializing these weights and updating and adjusting them with the data set to be trained. Once training is complete, when a new input is received, it infers an appropriate output value. The learning principle of an artificial neural network can be viewed as the process by which intelligence is formed from the generalization of experience, and it operates in a bottom-up manner. In Figure 1, when there are two or more intermediate layers (i.e., 5 to 10), the layers are considered to be deep, and it is called a Deep Neural Network; the learning and inference model achieved through such a Deep Neural Network can be referred to as Deep Learning.

[0063] Artificial neural networks can perform a certain role even with only one intermediate layer (commonly referred to as a 'hidden layer') in addition to inputs and outputs, but as the complexity of the problem increases, the number of nodes or layers must be increased. Among these, adopting a multi-layered model by increasing the number of layers is effective, but its scope of application is limited due to the limitations that efficient learning is impossible and the amount of computation required to train the network is large.

[0064] However, as the existing limitations mentioned above have been overcome, artificial neural networks have become capable of adopting deep structures. This has enabled the construction of complex and highly expressive models, leading to the 발표 of groundbreaking results in various fields such as speech recognition, face recognition, object recognition, and character recognition.

[0065] Figure 2 is a diagram illustrating an example of a deep neural network.

[0066] A Deep Neural Network (DNN) is an Artificial Neural Network (ANN) composed of multiple hidden layers between an input layer and an output layer. It is a set of machine learning models or algorithms referring to a Deep Neural Network (DNN) that has one or more hidden layers between an input layer and an output layer. Connections in a neural network are formed from the input layer to the hidden layer, and from the hidden layer to the output layer.

[0067] Deep neural networks, like general artificial neural networks, can model complex non-linear relationships. For example, in a deep neural network structure for an object identification model, each object can be represented as a hierarchical composition of the basic elements of an image. In this case, additional layers can combine features from progressively gathered lower layers. This characteristic of deep neural networks enables the modeling of complex data with fewer units (nodes) compared to similarly performed artificial neural networks.

[0068] Previous deep neural networks were typically designed as feedforward networks, but recent research has successfully applied deep learning structures to Recurrent Neural Networks (RNNs). Examples include the application of deep neural network structures in the field of language modeling. In the case of Convolutional Neural Networks (CNNs), not only have they been successfully applied in the field of computer vision, but their successful applications are also well-documented. More recently, CNNs have been applied to acoustic modeling for Automatic Speech Recognition (ASR) and are considered to have been more successful than existing models. Deep neural networks can be trained using the standard backpropagation algorithm. In this process, weights can be updated through stochastic gradient descent.

[0069] Deep learning using artificial neural networks is garnering significant attention across various fields as it far surpasses the performance of existing algorithms. However, currently used deep learning methods are difficult to apply in the resource-constrained mobile sector due to their high power consumption requirements. Consequently, there is growing interest in spiking neural networks (SNNs), which can operate at low power. SNNs learn synaptic weights using the STDP algorithm, which adjusts synaptic weights based on the time relationship between spikes before and after the synapse. Therefore, SNNs can learn in various configurations depending on the STDP algorithm, the number of spikes used for training, and the temporal interaction between spikes.

[0070] Neuromorphic chips, which are artificial intelligence computing chips, are attracting attention as a next-generation technology because they can solve the power supply issues of conventional semiconductor chips and integrate data processing processes. The core of neuromorphic technology lies in mimicking the human brain to enable the simultaneous execution of memory and computation on a massive scale. Furthermore, neuromorphic chips model how neurons in the brain communicate and learn by using synapses and spikes (electrical stimuli) that can be adjusted according to the situation. These chips are also designed to self-configure and make decisions by correlating learned patterns and associations.

[0071] Neuromorphic computing

[0072] The core challenge of neuromorphic research is to study the ability to learn from unstructured stimuli at an energy-efficient level comparable to that of the human brain, while rivaling human flexibility. The computing components of neuromorphic computing systems are logically similar to neurons. Spiking neural networks (SNNs) are a new model that arranges these elements to mimic the natural neural networks found in biological brains. Each "neuron" in an SNN operates independently of others, transmitting pulse signals to other neurons in the network and directly altering their electrical states. By encoding information and timing within the signals themselves, SNNs simulate natural learning processes by dynamically remapping synapses between artificial neurons in response to stimuli.

[0073] Neuromorphic computing, or neuromorphic engineering, is a field of engineering that aims to mimic human brain functions by creating circuits that imitate the shape of neurons. The circuits and chips created in this way are called neuromorphic circuits and neuromorphic chips, respectively. While artificial neural networks simulate the human nervous system through software, neuromorphic chips simulate nerve cells through hardware. In other words, a neuromorphic chip refers to a computer chip that mimics the structure of a biological nervous system (brain). Because it is a computer chip composed solely of circuits necessary for neural network computation, it offers advantages of hundreds of times or more in terms of power consumption, area, and speed compared to performing neural network calculations using CPUs and GPUs.

[0074] While dedicated ASIC chips for abstracted artificial neural network circuits, such as DNNs like Google's TPU or CNNs, exist, they are generally not classified as neuromorphic chips. The implementation of a TPU is similar to that of a general DSP, whereas widely researched neuromorphic chips typically implement individual neurons independently and possess higher data locality, usually utilizing learning algorithms other than backpropagation based on this. SNNs implementing Spike-Time Dependent Plasticity (STDP) are a representative example. Ultimately, this approach can offer better scalability and higher performance than conventional centrally controlled DSPs. However, due to difficulties in algorithm and circuit implementation, there have been no significant visible industrial achievements to date.

[0075] Unlike conventional computers, the human brain generates almost no power even when processing vast amounts of data. This is because the structure connecting neurons and synapses is arranged in parallel. Synapses conserve energy by connecting and disconnecting when they are working or not. While conventional computers consume a significant amount of electricity during the process of handling data between the CPU and memory, neuromorphic chips have reduced power consumption by mimicking the way the brain operates.

[0076] Spiking Neural Network (SNN) algorithm model

[0077] SNNs can be referred to as spiking artificial neural networks. The biggest difference from general artificial neural networks is the existence of a time axis. Instead of simply retrieving the values ​​of previous neurons once per neuron, the neuron's value (internal state) continuously changes over time. To prevent the network from becoming monotonous and to more closely mimic the actual brain, the concepts of thresholds and spikes were introduced. When a neuron's internal state value exceeds the threshold, it transmits a spike to connected neurons, and the internal state is reset. The neuron receiving the spike then increases or decreases its internal state depending on the synaptic weights. It is possible that receiving this spike may trigger another spike in the next neuron. After defining the time range, the input is provided as the "spikes of the input neuron," and the output is measured as the "number of spikes appearing in the output neuron."

[0078] Multi-Spiking Neural Network

[0079] Here, there is a model called the Multi-Spiking Neural Network that simulates the brain more precisely. In contrast to the previously introduced model, it is also referred to as the Single-Spiking Neural Network. The difference in this model is that multiple synapses connect the same pair of neurons. Each synapse has a different speed, and accordingly, the delay that occurs when a spike signal is transmitted varies.

[0080] The structure of the neural network itself is the same as a standard ANN. Simply put, it consists of an input layer, a hidden layer, and an output layer. While complex structures like Spiking CNNs could be drawn, we will leave them aside for now. All neurons in adjacent layers have been connected. Now, unit time (set in seconds) is divided into timestamps, and the process of constructing a new state from the neural network's state from one second ago is repeated as many times as the number of timestamps (usually 60). The state of the neural network consists of only a single variable: the internal state of the neuron, which will be discussed below. All other factors are either hyperparameters to be learned or constants.

[0081] Each neuron possesses a variable called "internal state" (V). This value starts at 0 by default and is a real value less than or equal to a threshold (Vth). Vth is a specific value assigned to each neuron and is one of the learning targets. However, input neurons do not possess this value and simply fire. The firing period is determined based on the corresponding input value (primarily using a Poisson distribution). Neurons that are not input neurons fire when V ≥ Vth (threshold voltage), and immediately after firing, V decreases by Vth. Mimicking the biological refractory period (the time during which a neuron that has fired once cannot fire again within a certain period), a neuron that has fired once does not fire again immediately.

[0082] The value of V continuously decreases (in absolute value) unless spikes (fires) are received from all neurons. In the paper, V is decreased in the form of an exponential function, decreasing by a factor of e1τ every second. If spikes are transmitted from neurons in all layers through synapses, V increases by the synaptic weight wij. Whether V increases or decreases is determined by the sign of w.

[0083] As such, SNNs can operate in an event-driven manner and enable low-power operation compared to other artificial neural networks because neurons fire only when their membrane potential is higher than a threshold voltage and transmit information between synapses through fired spikes. Since neurons and synapses in SNNs are not differentiable, SNNs cannot be trained using gradient descent or backpropagation. The most widely known training method for SNNs is Spiking Timing Dependent Plasticity (STDP). STDP is a method that learns synaptic weights through the temporal relationship between pre-synaptic and postsynaptic spikes. Therefore, the number of pre- and postsynaptic spikes considered in STDP training, as well as the temporal interaction between spikes, influences the learning of the SNN.

[0084] Figure 3 is a diagram illustrating the spike encoding method for input data when training a Spiking Neural Network (SNN) algorithm model.

[0085] Referring to Fig. 3, in order to train an SNN model, it is necessary to encode (or transform) the input data into a spike form. Spike encoding methods include rate coding, latency coding, and delta modulation, and the method of spike encoding varies depending on the input method. Spike encoding is possible for image data, 2D tensors, and 1D tensors (time series data).

[0086] Neuromorphic computing learning device

[0087] The SNN algorithm model includes an input neuron layer, a first neuron layer (corresponding to the excitatory neuron layer), and a second neuron layer (corresponding to the inhibitory neuron layer). In the present invention, the energy values ​​of the preprocessed UWB radar signal, which is the input data, can be input into the input neuron layer in the form of Leaky Integrate-and-Fire (LIF) after spike encoding. As shown in Fig. 4, the method of connecting the input neuron layer to the excitatory neuron layer in a fully-connected manner and learning the weights between them is the unsupervised learning algorithm Spike-Timing-Dependent Plasticity (STDP) algorithm. Spikes that fire beyond thresholds accumulate in the excitatory neuron layer, and the excitatory neuron layer is again connected one-to-one with the inhibitory neuron layer (second neuron layer) to transmit the fact of spike firing. That is, it transmits the spike. The inhibitory neuron layer is connected to the excitatory neuron layer in a many-to-many manner, and can transmit a negative (-) inhibitory value (e.g., -50) to all excitatory neurons except for the excitatory neuron that transmitted the spike (e.g., the first excitatory neuron). This is called lateral inhibition. As the series of processes is repeated, specific excitatory neurons that respond to specific data are determined.

[0088] LIF model

[0089] The Leaky Integrate-and-Fire (LIF) model is a neuron model used in SNNs designed to mimic the actual structure of neurons. In neurons, spikes received from multiple neurons via dendrites fire through the axon to other neurons when they exceed a threshold voltage. Similarly, in the LIF model, a threshold voltage called Vth is established; when the accumulated number of spikes (membrane potential) exceeds this threshold, spikes are fired to other neurons, and the membrane potential is reset by Vreset. Additionally, the neuron in question enters a refractory period, preventing it from firing spikes for several milliseconds. Since actual neurons receive spikes through channels called synapses, synapses are used as a concept akin to weights in the LIF model. A synapse is defined as the gap between the end of the axon—the projection of a neuron that conducts stimuli outside the cell—and the next neuron through which neurotransmitters travel.

[0090] Spike-Timing-Dependent Plasticity (STDP)

[0091] The Spike-Timing-Dependent Plasticity (STDP) algorithm is an unsupervised learning method in which the input neuron layer is fully connected to the excitatory neuron layer and the weights between them are learned. STDP is a learning method based on biological principles and is a representative unsupervised learning method for SNNs. The input neuron layer and the excitatory neuron layer are connected in a fully connected state, similar to the structure of an ANN. STDP is a method that learns each connection line, or synapse (=weight). Presynaptic and postsynaptic neurons are established based on the input neurons, and weights are adjusted by calculating the difference in spike firing times between the neurons. The closer the firing times are, the larger the absolute value of the weight increase or decrease; if the presynaptic and postsynaptic neurons fire sequentially, the weight increases, and if they fire in reverse order, it decreases. Through these established weight values, the relationship between the input neurons is recorded in the excitatory neuron layer as the number of fired spikes.

[0092] Inhibition weight update

[0093] Inhibition weight update is the process of adjusting inhibition weights (or inhibition values) to identify neurons that respond to the same data together. Neuromorphic computing modules can adjust inhibition weights to identify neurons that respond to the same data together.

[0094] In the excitatory neuron layer, the inhibitory neuron layer is connected one-to-one, for example, excitatory neuron A transmits a spike to inhibitory neuron A. In the inhibitory neuron layer, the excitatory neuron layer is connected many-to-many, so inhibitory neuron A can transmit a negative (-) value (e.g., -50) to all excitatory neurons except excitatory neuron A.

[0095] Figure 4 is a diagram illustrating the correlation between input data and output data during SNN model training according to the present invention.

[0096] Referring to Figures 4(a) and 4(b), a Leaky Integrate-and-Fire (LIF) neuron model is utilized as the activation function for training the SNN model after spike encoding. The non-linear activation of ReLU neurons in conventional ANNs and the firing rate of LIF neurons in SNN models exhibit a high correlation, and the SNN model can be trained based on these characteristics. If the amplitude value of 1D time-series data (e.g., energy of UWB radar signals, ECG signals, EEG signals) is large, the characteristics of the spikes appear differently depending on the spike encoding method. The energy value of the UWB radar signal is encoded and the corresponding spike is input into the SNN model to train the LIF model-based SNN, thereby displaying the output value in the form of a spike.

[0097] In the following, the present invention proposes a robot arm control algorithm for mimicking human hand and arm movements in medical and various industrial environments using robots, and a neuromorphic device for robot arm control.

[0098] In this invention, data obtained from human biosignals, such as electromyography (EMG) and a Dynamic Vision Sensor (DVS) camera, is classified using an artificial neural network called a Spiking Neural Network (SNN), thereby enabling low-power, fast, and precise control of a robot.

[0099] The SNN model proposed in this invention is a model inspired by neurobiology. Using this model offers the following advantages. Since the SNN is based on a model of neurons in the actual brain, it mimics the operating principles of the human nervous system. Because the SNN processes data through learning, it can learn new movements and respond to changes in the environment. This is useful when performing various tasks and helps implement intelligent movements. The SNN supports parallel processing, allowing it to process multiple sensor data simultaneously and respond quickly to various motion requests. This is important in real-time applications. Compared to conventional neural network models, the SNN is energy-efficient and suitable for battery-operated devices or wearable robots. Using the SNN allows for the real-time processing of EMG and DVS data to detect anomalies during arm movements. This helps the robotic arm operate safely and prevents malfunctions in advance.

[0100] In summary, when used in conjunction with EMG and DVS, SNN adds biological similarity and flexibility to robotic arm control, and offers technical advantages that make it useful in various applications through parallel processing and energy efficiency.

[0101] FIG. 5 is a diagram illustrating a block diagram for explaining the function and configuration of a neuromorphic device (500) according to the present invention.

[0102] Referring to FIG. 5, the neuromorphic device (500) according to the present invention may include a processor (510), a communication unit (520), and a memory (530).

[0103] The processor (510) performs computational processing (computing) on ​​input data according to the SNN (algorithm) model according to the present invention and outputs human hand and arm movements. The memory (530) stores various information necessary for neuromorphic computing, such as information necessary for the processor (510) to estimate results and information about the estimated results. The communication unit (520) receives information for computing in the processor (510) from an external device (ECG sensor, DVS camera, etc.) or transmits information to another external device.

[0104] FIG. 6 is a diagram illustrating a system for controlling a robot hand / arm by mimicking SNN-based hand / arm movements using an EMG sensor (100) and a DVS camera (200) according to the present invention.

[0105] Referring to FIG. 6, the EMG sensor (100) that can be worn by a person can be described as a wearable ECG sensor, and in the present invention, it is assumed that a person wears the EMG sensor (100) on their arm. The EMG sensor (100) can operate on the principle of detecting muscle movement by utilizing electromyographic characteristics, installing electrodes at specific sites, and recording electrical activity occurring within the muscle by detecting the potential difference between the two electrodes. The EMG sensor (100) is a bioelectrical signal generated during the muscle contraction process, and has the advantage of faster intention recognition than vision-based recognition, without problems of occlusion and distortion, and can estimate the muscle (force) intensity transmitted by the user. When the EMG sensor (100) detects a potential difference, it measures the amount of movement, and when it detects speed, it measures the nerve conduction velocity. The EMG sensor (100) senses the electromyogram signal of the person wearing it.

[0106] An example of the EMG sensor (100) in the present invention may be a Myo armband, a wearable device developed by Thalmic Labs that is worn on the arm and can detect arm movements and gestures using an electromyography (EMG) sensor. The EMG sensor (100) detects the movement and muscle activity of the user's arm and hand and transmits the information to another device (e.g., a communication unit (520), etc.). As described above, using the EMG sensor (100) allows for faster recognition of movement than vision-based recognition and eliminates problems of occlusion and distortion. However, since it is difficult to accurately determine the position information of the hand using the EMG sensor (100), a DVS camera (200) is used to capture the movement. The DVS camera (200) has a refresh rate of 1 µs, so it can capture minute movements. Additionally, since the DVS camera (200) is an event-based camera that captures only movement, it is suitable as an input for an SNN that operates based on spikes.

[0107] An event-based camera, as shown in FIG. 6, captures even slight changes in the movement of an object to be measured and outputs them as spike signals, even if there is only a minute brightness in the measurement environment. The DVS camera (200) is an event-based camera that mimics the way a human iris receives information and, unlike conventional frame-unit cameras, asynchronously tracks only the motion of moving objects. The DVS camera (200) detects the movements of a human hand or arm. Since the DVS camera (200) tracks only the events of moving pixels and outputs 1-bit spikes, it has a significant advantage over conventional cameras in terms of power consumption and data storage capacity. Because the DVS camera (200) has superior temporal resolution compared to high-speed cameras, it is suitable for real-time processing and is being widely utilized and researched in object recognition technology for autonomous vehicles that require fast judgment. The DVS camera (200) possesses the advantages of low power consumption, low capacity, and real-time processing.

[0108] SNN is nicknamed the third-generation neural network and is a machine learning technique aimed at mimicking the actual operation of the brain. Unlike existing artificial neural network techniques, it has the advantages of low power consumption and high efficiency because it only exchanges 1-bit spikes. Therefore, similar to DVS, it is suitable for edge devices or wearable devices and is being actively researched alongside neuromorphic hardware such as Intel’s Loihi, IBM’s True North, and Spinnaker. Since both the SNN and the DVS camera (200) exchange information in spike units, the captured data from the DVS camera (200) can be used as input data for the SNN algorithm model (or SNN model) without an additional conversion process.

[0109] In this way, the DVS camera (200) is an event-based camera that captures only movement, has a time refresh rate of 1 μs, and uses asynchronous communication, and can track location information that the EMG sensor (100) cannot capture.

[0110] The communication unit (520) can receive information regarding an electromyogram signal related to the movement of a person's hand or arm measured from the EMG sensor (100). The communication unit (520) can receive information regarding the movement of the person's hand or arm detected by the DVS camera (200). The processor (510) converts the information regarding the electromyogram signal received by the communication unit (520) into a spike signal.

[0111] The processor (510) applies the converted spike signal and information regarding the movement of a person's hand or arm received from the DVS camera (200) to a predetermined learned Spiking Neural Network (SNN) model. Classes based on movement and position may be defined in advance. Here, the classes are defined to distinguish the movement, position, and action of a person's hand or arm. The processor (510) outputs a class corresponding to the action of the person's hand or arm from among the predetermined classes using the predetermined learned SNN model.

[0112] The communication unit (520) can transmit the output class to a robot (e.g., a prosthetic arm robot, a surgical robot, an explosives disposal robot) (300) to reproduce the movement of the human hand or arm by imitating it. The robot (300) can reproduce the movement of the human hand or arm corresponding to the class received from the communication unit (520) by imitating it exactly.

[0113] FIG. 7 is a diagram illustrating the process of a neuromorphic device (500) according to the present invention converting or encoding ECG data into a spike signal.

[0114] Referring to FIG. 7, the EMG sensor (100) can measure information regarding an electromyogram signal related to the movement of a person's hand or arm. Here, the information regarding the measured electromyogram signal includes electromyogram data of the person's hand or arm.

[0115] The processor (510) can obtain information regarding the position data of a person's hand or arm and electromyography data from the EMG sensor (100). The processor (510) can extract features from the electromyography data of the person's hand or arm using an adaptive filter. After extracting the features, the processor (510) can perform spike transformation or encoding. Here, the features (values) extracted using the adaptive filter may be as follows, as an example.

[0116] Feature values ​​in EMG signals

[0117] 1. Mean: Represents the average signal strength of the electromyography signal.

[0118] 2. Standard deviation: Represents the standard deviation of the electromyography signal intensity.

[0119] 3. Zero Crossing: Represents the number of zero crossings when the electromyogram signal is second-order differentiated.

[0120] 4. Kurtosis: A value obtained by normalizing the fourth momentum of a signal by the square of its standard deviation, representing a value that determines the peakedness of the probability distribution of signal strength.

[0121] 5. Crest factor: The ratio of the signal's peak values. Similar to Kurtosis, it is a characteristic value that describes the impulseness of the signal.

[0122] 6. Power Spectrum Density: Represents the power spectrum density generated through the Fourier transform of the electromyography signal.

[0123] 7. Correlation: Represents the auto-correlation of the electromyography signal.

[0124] 8. Threshold Crossing: Represents the number of filtered raw data that exceed a specific threshold in one epoch.

[0125] 9. Skewness: As the third-order momentum of the electromyogram signal, it indicates the degree of bias in the distribution of signal intensity.

[0126] 10. Entropy: Represents a measure of the predictability of a signal.

[0127] 11. Band Energy: Represents energy in the frequency range associated with concentration.

[0128] 12. Spectral Flux: Refers to the rate of change of the power spectrum over time during time-frequency analysis of a signal (PSD(t) / PSD(t-1))

[0129] FIG. 8 is a diagram illustrating the processing steps performed in the DVS camera (200).

[0130] Referring to FIG. 8, the DVS camera (200) captures and acquires DVS data, undergoing a process of cropping, down-sampling, and framing. As described above, since both the SNN and the DVS camera (200) exchange spike unit information, the captured data of the DVS camera (200) can be used as input data to the SNN algorithm model (or SNN model) without an additional conversion process.

[0131] FIG. 9 is a diagram illustrating the result of implementing a neuromorphic device (500) according to the present invention to mimic and reproduce the movements of a real person's hand and arm.

[0132] Referring to Figure 9, the SNN model may be a model trained through unsupervised learning, and about 6 classes were predefined.

[0133] After the processor (510) performs processing and spike conversion on the input values ​​received from the EMG sensor (100) and applies them to the trained SNN model, the results for the output classes show a high accuracy of 82% classification accuracy. When examining the results for the output classes after applying the results received from the DVS camera (200) to the trained SNN model, it can be seen that the accuracy reaches 87%.

[0134] FIG. 10 is a drawing comparing the result of implementing the neuromorphic device (500) according to the present invention to mimic and reproduce the movements of a real person's hand and arm with the prior art.

[0135] Referring to FIG. 10, the neuromorphic device (500) according to the present invention may be a spiking CNN model, and in this case, it shows higher accuracy than using a CNN (CPU) model, has a power reduction effect of about 30 times in terms of energy, and has an effect of improving speed (inference speedup) by 30 times.

[0136] Along with the advancement of artificial intelligence, the field of Human Robot Interaction is also continuously developing. However, most AI systems currently in widespread use utilize cloud servers to process user data. The primary reason for using cloud servers is that there are still many hardware limitations preventing the direct integration of models into devices. Nevertheless, with the emergence of the Spiking Neural Network, a next-generation neural network proposed in this invention, AI can directly experience the environment and process information on edge devices—the hardware that interacts directly with the user—thereby enabling the utilization of advantages that allow for decisions more suitable for the actual user. Therefore, this invention can achieve the following effects.

[0137] First, accurate motion imitation is possible. The SNN algorithm model proposed in this invention utilizes EMG and DVS data to enable a robot arm to precisely imitate human arm movements. As a result, the robot can accurately perform fine movements and complex actions, making it suitable for use in medical surgery, assembly lines, or other sophisticated tasks.

[0138] Second, it possesses real-time responsiveness. Since the present invention, based on SNN, processes EMG and DVS data in real time, the robot arm responds quickly to user movements. This is an important feature in work environments where rapid response is required.

[0139] Third, anomaly detection and enhanced safety are possible. The present invention improves safety by detecting anomalies during arm movements and preventing malfunctions in advance. This helps ensure the safe operation of the robot during medical surgery or in hazardous environments.

[0140] Fourth, diverse application fields: This technology can be utilized in various application fields such as medical, industrial automation, assistive devices for the disabled, robotics, and security, and can lead innovation in these fields.

[0141] Fifth, continuous learning and collaboration are possible. By utilizing SNNs, robots can continuously learn and improve their performance through interaction with users. This enables the robot to understand and improve various tasks. Consequently, this invention enhances the performance, accuracy, and safety of robot arms and expands their potential for real-world application by providing innovative robot control solutions in various fields.

[0142] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0143] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computing devices and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0144] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0145] The embodiments described above are combinations of the components and features of the present invention in a specific form. Each component or feature should be considered optional unless otherwise explicitly stated. Each component or feature may be implemented in a form not combined with other components or features. Additionally, it is possible to construct embodiments of the present invention by combining some components and / or features. The order of operations described in the embodiments of the present invention may be changed. Some components or features of one embodiment may be included in another embodiment, or may be replaced with corresponding components or features of another embodiment. It is obvious that embodiments may be constructed by combining claims that do not have an explicit citation relationship in the claims, or that new claims may be included by amendment after filing.

[0146] In the present invention, the processor (510) may be implemented by hardware, firmware, software, or a combination thereof. When implementing an embodiment of the present invention using hardware, ASICs (application specific integrated circuits) or DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), etc. configured to perform the present invention may be provided in the processor (510).

[0147] It is obvious to those skilled in the art that the present invention may be embodied in other specific forms without departing from the essential features of the invention. Accordingly, the foregoing detailed description should not be interpreted restrictively in all respects but should be considered exemplary. The scope of the invention shall be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the invention are included within the scope of the invention.

[0148] An SNN-based arm motion mimicking robot arm control method using EMG and DVS and a neuromorphic device for this purpose are industrially available to control robot motion by precisely and intricately mimicking hand and arm movements with low power.

Claims

1. Receive information regarding electromyography signals related to the movement of a person's hand or arm measured from an EMG (Electromyography) sensor, and A communication unit that receives information about the movement of the person's hand or arm detected by a DVS (Dynamic Vision Sensor) camera; Converting information regarding the received electromyography signal into spike signals, A neuromorphic device for mimicking and reproducing a human hand or arm movement, characterized by including a processor that applies the converted spike signal and information about the human hand or arm movement received from the DVS camera to a predetermined learned Spiking Neural Network (SNN) model to output a class corresponding to the movement of the human hand or arm among classes defined in advance based on movement and position.

2. In Paragraph 1, A neuromorphic device for reproducing human hand or arm movements, characterized in that the above-mentioned communication unit transmits the above-mentioned output class to a robot to reproduce the human hand or arm movements by imitating them.

3. In Paragraph 1, The above processor is, Acquire electromyography data of the hand or arm of the person mentioned above, and A neuromorphic device for mimicking and reproducing human hand or arm movements, characterized by extracting features from electromyography data of the human hand or arm using an adaptive filter and then performing spike transformation.

4. In Paragraph 3, A neuromorphic device for mimicking and reproducing human hand or arm movements, characterized in that the processor extracts features using the adaptive filter and then performs spike transformation using delta-sigma modulation.

5. A step of receiving information about an electromyography signal related to the movement of a person's hand or arm measured from an EMG sensor; A step of receiving information about the movement of the person's hand or arm detected by a DVS (Dynamic Vision Sensor) camera; A step of converting information about the received electromyogram signal into a spike signal; The step of inputting the converted spike signal and information regarding a person's hand or arm movement received from the DVS camera into a predetermined learned Spiking Neural Network (SNN) model; and A method for a neuromorphic device to mimic and reproduce a human hand or arm motion, characterized by including the step of outputting a class corresponding to the motion of the human hand or arm among classes defined in advance based on movement and position.

6. In Paragraph 5, A method for a neuromorphic device to reproduce a human hand or arm motion by imitating it, characterized by further including the step of transmitting the above-mentioned output class to a robot to reproduce the above-mentioned human hand or arm motion by imitating it.

7. In Paragraph 5, The step of converting to the above spike signal is, A step of acquiring electromyography data of the hand or arm of the person; and A method for a neuromorphic device to mimic and reproduce human hand or arm movements, characterized by including the step of extracting features from electromyography data of the human hand or arm using an adaptive filter and then performing spike transformation.

8. In Paragraph 7, A neuromorphic device for mimicking and reproducing human hand or arm movements, characterized in that the above spike transformation step involves transforming spikes using delta-sigma modulation.

9. An EMG sensor that measures electromyography signals related to the movement of a person's hand or arm; A DVS (Dynamic Vision Sensor) camera that detects the movement of the hand or arm of the person mentioned above; and The information regarding the electromyogram signal measured by the EMG sensor is converted into a spike signal, and A neuromorphic-based system for mimicking and reproducing human hand or arm movements, characterized by including a processor that applies the converted spike signal and information regarding the human hand or arm movement output by the DVS camera to a predetermined learned Spiking Neural Network (SNN) model to output a class corresponding to the movement of the human hand or arm among classes defined in advance based on movement and position.

10. A computer-readable recording medium storing a program for executing the method described in any one of paragraphs 5 through 8 on a computer.

Images