Brain-computer interface device and control method therefor, medium, and electronic device

Abstract

The present application provides a brain-computer interface device and a control method therefor, a medium, and an electronic device. The control method for a brain-computer interface device comprises: acquiring a brain signal generated by a subject in any stimulus state; performing feature recognition on the brain signal, and obtaining a brain response feature included in the brain signal; performing instruction recognition on the brain signal according to the brain response feature, and obtaining a target instruction represented by the brain signal; and configuring a preset execution probability on the basis of the target instruction, and outputting the target instruction on the basis of the execution probability so as to control an external controlled device. In the present application, by performing response feature recognition on a brain signal and performing instruction recognition according to the response feature, a target instruction can be recognized from the brain signal in real time, and can be output in real time to control an external controlled device, thereby meeting the requirements of application fields having high real-time demands.

Description

Brain-computer interface devices, control methods, media and electronic devices thereof Technical Field

[0001] This application belongs to the field of brain-computer interface technology, and relates to a brain-computer interface device and its control method, medium and electronic device. Background Technology

[0002] Brain-computer interface (BCI) is a technology that establishes direct communication between the human brain and external devices. It bypasses traditional neural and muscular pathways, allowing brain activity to directly control external devices. Control systems based on non-invasive BCIs have already enabled operations such as character spelling, robotic arm control, and drone control, attracting widespread attention. Non-invasive BCIs offer advantages such as wider applicability, fewer ethical concerns, and non-invasiveness.

[0003] However, existing BCI systems rely on acquiring weak electrical signals from the surface of the brain, which are easily affected by external noise. Advanced signal processing techniques are needed to improve the accuracy and stability of the signals. Signal processing takes a lot of time, resulting in slow command output, which cannot meet the needs of practical applications. Summary of the Invention

[0004] This application provides a brain-computer interface device and its control method, medium and electronic device to solve the problem that the instruction output of existing brain-computer interface devices is slow and cannot meet the needs of application fields with high real-time requirements.

[0005] In a first aspect, this application provides a control method for a brain-computer interface device, comprising: acquiring brain signals generated by a subject under any stimulation state; performing feature recognition on the brain signals to obtain brain response features contained in the brain signals; performing instruction recognition on the brain signals under the brain response features to obtain a target instruction represented by the brain signals; configuring a preset execution probability based on the target instruction; and outputting the target instruction based on the execution probability to control an external controlled device.

[0006] In this implementation, by recognizing the response features of brain signals and the instructions under the response features, the target instructions can be identified from the brain signals in real time, and the target instructions can be output in real time to control the external controlled devices, which can meet the needs of application fields with high real-time requirements.

[0007] In one implementation of the first aspect, the execution probability of the identified target instruction is calculated iteratively based on a Kalman filter; each time the target instruction is identified, the corresponding iterative calculation increases by one generation; it is determined whether the execution probability of the target instruction is greater than or equal to a preset value; if so, the target instruction is output to control an external controlled device; if not, the target instruction is not output.

[0008] In this implementation, by using a Kalman filter to set the execution probability control of the target instruction output, the target instruction is smoothly processed, which can greatly reduce the false trigger rate. In addition, it can not only improve the accuracy of target instruction execution and / or termination, but also precisely control the delay of target instruction line and / or termination.

[0009] In one implementation of the first aspect, it is determined whether the currently identified target instruction is the same as the previously identified target instruction; if they are the same, the execution probability of the currently identified target instruction increases; if they are different, the execution probability of the currently identified target instruction decreases; and the execution probability of the target instruction is updated to the execution probability of the currently identified target instruction.

[0010] In this implementation, by adjusting the execution probability of the target instruction by increasing or decreasing it, the accuracy and effectiveness of the target instruction output can be further improved. Each target instruction can be set with a separate increase / decrease adjustment strategy, which can further match the actual control scenario of the external controlled device. It can be flexibly configured according to the actual situation of the external controlled device, and the effectiveness of the brain-computer interface control device is not affected by the changes of the external controlled device. It has a wide range of applications and stronger versatility.

[0011] In one implementation of the first aspect, the accuracy of the execution and / or termination of the target instruction is controlled based on a probability function; the delay of the execution and / or termination of the target instruction is controlled based on the calculation parameters of the probability function; the probability function includes an increasing probability function and a decreasing probability function; the execution probability of the current target instruction is increased based on the increasing probability function; the execution probability of the current target instruction is decreased based on the decreasing probability function; each target instruction has its own probability function.

[0012] In this implementation, by setting the probability function and calculation parameters, the problem of "accidental touch" in the target instruction output in practical applications can be solved.

[0013] In one implementation of the first aspect, the brain response features include visual brain response features and non-visual brain response features; the non-visual brain response features include muscle activity brain response features and resting-state brain response features; the target instruction recognition model identifies the brain signal based on the visual brain response features to obtain operation-type instructions and / or movement-type instructions represented by the brain signal; the target instruction recognition model identifies the brain signal based on the muscle activity brain response features to obtain movement-type instructions and / or control-type instructions represented by the brain signal; the target instruction recognition model identifies the brain signal based on the resting-state brain response features to obtain resting-state instructions represented by the brain signal; wherein, the resting-state instructions refer to instructions that do not execute the operation-type instructions, the movement-type instructions, and the control-type instructions; the control-type instructions enable, stop, or / and switch the execution of other types of instructions.

[0014] In this implementation, operational and / or movement commands can be obtained based on the visual brain response feature recognition, movement and / or control commands can be obtained based on the muscle activity brain response feature recognition, and resting-state commands can be obtained based on the resting-state brain response feature recognition. By recognizing the muscle activity brain response features and the resting-state brain response features, the number of target commands corresponding to muscle activity and resting-state can be increased, further increasing the number of commands and thus expanding the degree of freedom of system control.

[0015] In one implementation of the first aspect, the resting state instruction further includes retaining the original action of the previous instruction under the condition that the operation instruction, the movement instruction and the control instruction are not executed.

[0016] In this implementation, the resting state instruction is a very important type of instruction. It can be implemented in conjunction with other types of instructions to improve the execution effect of other types of instructions and alleviate visual control fatigue of the subject.

[0017] In one implementation of the first aspect, the subject's visual direction signal is acquired based on an inertial measurement unit; the visual direction signal is used for instruction recognition to obtain that the target instruction represented by the visual direction signal is a visual direction instruction and / or a direction instruction; the visual direction instruction and / or direction instruction can be executed simultaneously with the operation instruction, movement instruction, control instruction and / or resting state instruction.

[0018] In this implementation, the perspective direction signal collected by the inertial measurement unit can be introduced to further enrich the dimensions of the target command and realize more degrees of freedom in the control of the target command.

[0019] In one implementation of the first aspect, the brain signal is identified based on a response feature recognition model to obtain brain response features contained in the brain signal, including: acquiring a training dataset, the training dataset including brain signals generated by the subject under various stimulus states; extracting features from the training dataset to obtain a response feature template dataset; and inputting the response feature template dataset and the brain signal into the response feature recognition model for processing to obtain the brain response features of the brain signal.

[0020] In one implementation of the first aspect, the response feature recognition model includes a first spatial filter, a correlation coefficient unit, and a response feature output unit; the first spatial filter filters a preset response feature template dataset to obtain a response feature filtered dataset; the first spatial filter filters the brain signal to obtain a first filtered feature; the correlation coefficient unit calculates the correlation value between the filtered feature and each response feature filtered data in the response feature filtered dataset, and determines that the response feature corresponding to the response feature filtered data with the largest correlation value is the brain response feature of the brain signal; the response feature output unit outputs the brain response feature.

[0021] In this implementation, the target response features in the brain signal are accurately identified by the response feature recognition model, while non-target response features are suppressed, thus providing a basic reference for subsequent more accurate target instruction recognition.

[0022] In one implementation of the first aspect, a target instruction recognition model is used to identify the brain signal under the brain response features to obtain the target instruction represented by the brain signal; the target instruction recognition model includes a second spatial filter, a correlation processing unit, a classifier, and an instruction output unit; the second spatial filter filters the response feature template data corresponding to the brain response features to obtain corresponding response feature filtered data; the second spatial filter filters the brain signal to obtain second filtered data; the correlation processing unit calculates the coefficient vector of the second filtered data and the response feature filtered data; the classifier classifies the coefficient vector to identify the target instruction represented by the brain signal; and the instruction output unit outputs the target instruction.

[0023] In this implementation, based on the accurate identification of target response features in brain signals by the response feature recognition model, the target instruction in brain signals is further identified more accurately by the target instruction recognition model, which improves the accuracy of instruction recognition and also greatly increases the information transmission rate of target instructions.

[0024] In one implementation of the first aspect, response feature analysis is performed on the acquired training dataset to obtain a spatial filter; the response feature analysis includes task discrimination component analysis, task-related component analysis, or canonical correlation analysis.

[0025] Secondly, this application provides a brain-computer interface device, comprising: an input interface module for acquiring brain signals generated by a subject under any stimulus state; a response feature recognition model for recognizing response features of the brain signals to obtain stimulus response features contained in the brain signals; a target instruction recognition model for recognizing instructions under the response features of the brain signals to obtain target instructions represented by the brain signals; and an output interface module for outputting the target instructions to control external controlled devices.

[0026] Thirdly, this application provides an electronic device, including: one or more processors; and one or more memories, wherein the memories store computer-readable code, which, when executed by the one or more processors, implements the control method of the brain-computer interface device as described above.

[0027] Fourthly, this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the control method for the brain-computer interface device as described above. Attached Figure Description

[0028] Figure 1 shows an exemplary application scenario of the brain-computer interface-based control method described in the embodiments of this application.

[0029] Figure 2 shows an exemplary flowchart of a control method for a brain-computer interface device according to an embodiment of this application.

[0030] Figure 3A shows an exemplary flowchart of step S202 of the control method described in an embodiment of this application.

[0031] Figure 3B shows an exemplary result diagram of the control method described in Figure 3A.

[0032] Figure 3C shows another exemplary flowchart of step S202 of the control method described in this application embodiment.

[0033] Figure 4A-1 shows an exemplary flowchart of step S204 of the control method described in an embodiment of this application.

[0034] Figure 4A-2 shows a schematic diagram of the three-state Kalman filter described in an embodiment of this application.

[0035] Figure 4A-3 shows a schematic diagram of the 8-state Kalman filter described in an embodiment of this application.

[0036] Figure 4A-4 shows a schematic diagram of brain signal filtering during a specific brain response period as described in an embodiment of this application.

[0037] Figure 4B-1 shows an exemplary flowchart of step S410 of the control method described in an embodiment of this application.

[0038] Figure 4B-2 shows an exemplary schematic diagram of the increase / decrease in execution probability as described in an embodiment of this application.

[0039] Figures 4C-1 and 4C-2 show exemplary schematic diagrams of the execution mode of the target instruction described in the embodiments of this application.

[0040] Figure 5 shows another exemplary flowchart of the control method for the brain-computer interface device described in the embodiments of this application.

[0041] Figure 6 shows an exemplary structural diagram of the response feature recognition model described in an embodiment of this application.

[0042] Figure 7 shows an exemplary structural diagram of the target instruction recognition model described in an embodiment of this application.

[0043] Figure 8A shows an exemplary flowchart of the online decoding analysis and recognition method for brain signals described in an embodiment of this application.

[0044] Figure 8B shows a schematic diagram of one identification result of the method described in Figure 8A.

[0045] Figure 8C shows a schematic diagram of another identification result of the method described in Figure 8A.

[0046] Figure 9 shows an exemplary structural diagram of the brain-computer interface device described in an embodiment of this application.

[0047] Figure 10 shows an exemplary structural diagram of an electronic device according to an embodiment of this application.

[0048] Component Labeling Description: 110 External Stimulation Module; 120 Brain Signal Acquisition Module; 130 Brain Signal Processing Module; 140 Conversion Control Module; 150 External Controlled Device; 600 Response Feature Recognition Model; 610 First Spatial Filter; 620 Correlation Coefficient Unit; 630 Response Feature Output Unit; 700 Target Command Recognition Model; 710 Second Spatial Filter; 720 Correlation Processing Unit; 730 Classifier; 740 Command Output Unit; 900 Brain-Computer Interface Device; 910 Input Interface Module; 920 Response Feature Recognition Model; 930 Target Command Recognition Model; 940 Output Interface Module; 100 Electronic Equipment; S201~S204; Steps S301~S304; Steps S410~S440; Steps S411~S414; Steps S501~S503; Steps... Detailed Implementation

[0049] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0050] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0051] Brain-computer interface (BCI) is a technology that establishes a direct connection between the human or animal brain and an external controlled device (such as a computer or robot), enabling information exchange and functional integration between the nervous system and the external controlled device. BCI technology includes invasive and non-invasive BCIs. Invasive BCIs acquire signals by implanting electrodes inside the brain; this method can obtain more precise neural signals, but it carries higher safety risks and technical barriers. Non-invasive BCIs record brain activity from the surface of the head using a scalp-worn device, eliminating the need for surgery and device implantation. This technology is safer, reduces the risks of brain surgery, and can be used to treat various neurological diseases and manage biological health. However, existing BCI systems suffer from slow command output and accidental command input, limiting their practical application in related fields.

[0052] The technical solutions in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0053] Figure 1 shows a schematic diagram of an application scenario of this application. The brain-computer interface-based control scenario includes: an external stimulation module 110, a brain signal acquisition module 120, a brain signal processing module 130, a conversion control module 140, and an external controlled device 150.

[0054] The external stimulation module 110 refers to the object that outputs specific stimuli, and is not limited to the environment, devices, and / or equipment. The types of stimuli include physical stimuli such as sound, touch, smell, and vision, as well as mental stimuli such as emotions, thoughts, and feelings.

[0055] The brain signal acquisition module 120 is configured to acquire brain signals generated by a subject (human or animal) under specific stimuli. Brain signals include electroencephalogram (EEG), magnetic resonance imaging (MEG), functional magnetic resonance imaging (fMRI), functional near-infrared spectroscopy (fNIRS), and / or neuronal spike potentials and local field potentials (LFP). Among these, EEG is the spatiotemporal characteristic electrical activity of the brain formed by the synchronous discharge of local groups of neurons in the brain. Based on frequency, EEG can be divided into the following main categories: delta waves (0.5–3 Hz), mainly appearing during deep sleep; theta waves (4–7 Hz), associated with meditation, drowsiness, hypnosis, or sleep; alpha waves (8–13 Hz), mainly appearing when awake, quiet, and with eyes closed; beta waves (13–34 Hz), associated with active, busy mental activity; and gamma waves (35 Hz and above), associated with cognitive processing and perception. Magnetoencephalography (MEG) is a signal that reflects changes in the brain's magnetic field, providing similar temporal resolution to EEG but with higher spatial resolution. Functional magnetic resonance imaging (fMRI) indirectly reflects neural activity by measuring changes in blood oxygen levels caused by brain activity, offering high spatial resolution but relatively low temporal resolution. Functional near-infrared spectroscopy (fNIRS) monitors neural activity by measuring changes in hemoglobin concentration caused by brain activity; it is a non-invasive brain imaging technique. Neuronal spike potentials and local field potentials (LFPs) are commonly used signal types in invasive or semi-invasive brain-computer interfaces, providing more direct information about neuronal activity.

[0056] For example, an interface can be used to present SSVEP stimuli of various frequencies (8.5-12 Hz, with a step size of 0.5 Hz). While the subject gazes at the interface, the brain signal acquisition module collects their physiological electrical signals (collectively referred to as brain signals). When the subject gazes at a specific frequency of the interface, the brain produces a corresponding response, which is manifested in the brain signal as an increase in the amplitude of the specific frequency component. The response amplitude refers to the intensity of the specific frequency component in the brain signal, and it reflects the intensity of the subject's response to the stimulus.

[0057] The brain signal processing module 130 is configured to process and analyze the acquired brain signals, extract effective features from the brain signals, and identify the effective features to obtain brain intention information for subsequent applications.

[0058] The brain control module 140 is configured to convert and process the brain intention information recognized by the brain signal processing module 130 into specific intention commands, and control the external controlled device 150 through the intention commands, causing the external controlled device 150 to execute the actions corresponding to the intention commands. The brain control module outputs a specific command, and controls the external controlled device (such as a computer terminal device or a game terminal device) through the specific command to control the virtual keyboard or gamepad.

[0059] As shown in Figure 2, this application provides a control method for a brain-computer interface device, including the following steps S201 to S204.

[0060] S201, acquire brain signals generated by the subject under any stimulus state.

[0061] In one implementation of this embodiment, the brain signal can be generated based on the BCI paradigm. The BCI paradigm refers to a set of specific psychological tasks or external stimuli carefully selected / designed by BCI developers under specific brain imaging technology to represent the intention of the subject or user (hereinafter referred to as the user).

[0062] In one implementation of this embodiment, the stimulus state includes any one or more specific stimuli such as physical stimuli like sound, touch, smell, and / or vision, and mental stimuli like emotions, thoughts, and feelings. Brain signals are formed after responding to specific stimuli; therefore, brain signals contain response characteristics corresponding to those specific stimuli.

[0063] S202, perform feature recognition on the brain signal to obtain the brain response features contained in the brain signal.

[0064] In one implementation of this embodiment, the brain response features may be frequency response features, phase response features, and / or amplitude response features, etc. The brain response features include visual brain response features and non-visual brain response features; the non-visual brain response features include muscle activity brain response features, resting-state brain response features, or other brain-like response features.

[0065] S203, perform instruction recognition on the brain signal under the brain response features to obtain the target instruction represented by the brain signal.

[0066] In one implementation of this embodiment, if the brain response feature is a visual brain response feature, the recognized instruction can include at least 8 types; if the brain response feature is a muscle activity brain response feature, the recognized instruction can include at least 5 types, such as clenching teeth, blinking, closing eyes, coughing, or resting; if the brain response feature is a resting-state brain response feature, the recognized instruction includes at least 1 type, such as a resting-state instruction. Based on the above three types of brain response features, instruction encoding with at least 14 degrees of freedom can be achieved.

[0067] S204, configure a preset execution probability based on the target instruction, and output the target instruction based on the execution probability to control an external controlled device.

[0068] This embodiment extracts the response features of brain signals to specific stimuli, identifies the target instructions reflecting the brain's intent within these response features, and outputs the target instructions to control external controlled devices. By recognizing the response features of brain signals and the instructions derived from those features, this embodiment can identify target instructions from brain signals in real time and output filtered target instructions to control external controlled devices in real time, meeting the needs of applications with high real-time requirements.

[0069] In one embodiment of this application, the brain response features include visual brain response features and non-visual brain response features; the non-visual brain response features include muscle activity brain response features and resting state brain response features. As shown in FIG3A, the control method of the brain-computer interface device described in this application embodiment may further include steps S301 to S303.

[0070] S301, based on the visual brain response features, identify the brain signal to obtain the operation-type instructions and / or movement-type instructions represented by the brain signal.

[0071] S302, based on the muscle activity brain response characteristics, the brain signal is identified to obtain the movement command and / or control command represented by the brain signal; wherein, the target command corresponding to the muscle activity brain response characteristics includes a preset command obtained from any non-visual evoked action such as clenching teeth, blinking, closing eyes, coughing, or resting; wherein, each non-visual evoked action corresponds to a preset command.

[0072] S303, based on the resting-state brain response characteristics, the brain signal is identified to obtain the resting-state instruction represented by the brain signal; wherein, the resting-state instruction refers to an instruction that does not execute the operation instruction, the movement instruction, and the control instruction; the control instruction enables the start, stop, or / and switch of the execution of other instruction types.

[0073] In this embodiment, operational commands and / or movement commands can be obtained based on the visual brain response feature recognition, movement commands and / or control commands can be obtained based on the muscle activity brain response feature recognition, and resting state commands can be obtained based on the resting state brain response feature recognition. Operational commands include commands for actions such as walking and attacking; movement commands include commands for moving in various directions such as forward, backward, left, and right; control commands include commands for control such as start, stop, and switch; and resting state commands are equivalent to no commands.

[0074] For example, Figure 3B of the embodiment provides a method for recognizing brain signals based on visual brain response features, muscle activity brain response features, and resting-state brain response features. The method includes: collecting 2 seconds of real-time brain signals (step size 20 milliseconds) from a brain signal acquisition terminal. After collecting the brain signals, response feature recognition is performed. If a resting-state brain response feature is identified, a resting-state command (i.e., a target command with specific features) can be directly output. If a muscle activity response feature is identified, a muscle activity (e.g., teeth clenching) command (i.e., a target command with specific features) can be output. If a visual brain response feature is identified, it can be determined as an SSVEP command, and then a specific command among the multiple commands contained in the SSVEP command is further identified through a multi-command recognition module (i.e., the target command recognition model described below). The target command is decoded into a specific external controlled device execution command via a virtual keyboard or gamepad.

[0075] For example, if the target command is applied to game control, and the external controlled device is a game device, then the resting state command can be used to control the game character to continue performing the previous action, such as walking; the bite command can be used to control the opening and closing of the entire game control system, or to control the start, stop or switch of the target command execution; and the SSVEP command can be used to control the internal operations of the game, such as: light attack, heavy attack, roll, jump, recovery, skill 1 / 2 / 3, etc.

[0076] In this implementation, operational and / or movement commands can be obtained based on the visual brain response feature recognition, movement and / or control commands can be obtained based on the muscle activity brain response feature recognition, and resting-state commands can be obtained based on the resting-state brain response feature recognition. By recognizing the muscle activity brain response features and the resting-state brain response features, the number of target commands corresponding to muscle activity and resting-state can be increased, further expanding the degrees of freedom of control, and also increasing the number of target commands that can be realized by the visual brain response features.

[0077] In one embodiment of this application, the resting state instruction further includes retaining the original action of the previous instruction without executing the operation instruction, the movement instruction, and the control instruction.

[0078] In this implementation, the resting-state command is a crucial command type. It can be implemented in conjunction with other command types, enhancing the execution effectiveness of those commands while alleviating visual fatigue in the user. Implementing resting-state commands significantly improves control fluency, reduces visual strain, allows the continuation of the previous action, and provides a brief rest before command switching. For example, in game control, if no other command is recognized during an attack, the attack can continue as a resting-state command and will not be stopped; or if a command gap occurs during a transition, resulting in a stuck, outputless state, the resting-state command can smoothly bridge the intermediate state and connect to the next command state, instead of passively stopping the attack.

[0079] In one embodiment of this application, as shown in FIG3C, the control method of the brain-computer interface device described in this embodiment may further include step S304, acquiring the subject's visual direction signal based on the inertial measurement unit, performing instruction recognition on the visual direction signal, and obtaining the target instruction represented by the visual direction signal as a visual direction instruction and / or a direction instruction; the visual direction instruction and / or direction instruction may be executed simultaneously with the operation instruction, movement instruction, control instruction and / or resting state instruction.

[0080] In one embodiment of this application, the inertial measurement unit includes a three-axis inertial measurement unit and a six-axis inertial measurement unit. The three-axis inertial measurement unit (IMU) is used to detect and measure the motion state of an object along three spatial axes (typically X, Y, and Z axes), including acceleration, tilt, impact, vibration, rotation, and multi-degree-of-freedom motion. An IMU typically includes a three-axis gyroscope, a three-axis accelerometer, and a three-axis magnetometer, used to measure the object's angular velocity, acceleration, and orientation information, respectively. The six-axis inertial measurement unit (IMU) is a sensor integrating a three-axis accelerometer and a three-axis gyroscope, used to measure the object's acceleration and angular velocity in space.

[0081] For 2.5D / 3D games that require perspective switching, this application can also receive and process gyroscope data and convert it into virtual keyboard / gamepad signals to achieve perspective switching.

[0082] In this implementation, the perspective direction signal collected by the inertial measurement unit can be introduced to further enrich the dimensions of the target command and realize more degrees of freedom in the control of the target command.

[0083] In one embodiment of this application, as shown in FIG4A-1, step S204 described in this embodiment may further include steps S410 to S440.

[0084] S410, the execution probability of the identified target instruction is calculated iteratively based on the Kalman filter; each time the target instruction is identified, the corresponding iterative calculation increases by one generation;

[0085] S420, determine whether the execution probability of the target instruction is greater than or equal to a preset value;

[0086] S430, if so, output the target instruction to control the external controlled device;

[0087] S440, if not, then the target instruction will not be output.

[0088] In this implementation, since resting state commands are not easy to recognize and command mis-touch is easy to occur when the gaze shifts, this embodiment uses a Kalman filter to set the execution probability control of the target command output, so as to achieve smooth processing of the target command. This can greatly reduce the mis-touch rate, and not only can it improve the accuracy of target command execution and / or termination, but it can also precisely control the delay of target command line and / or termination. This allows the technical solution described in this application to effectively achieve precise control of external controlled devices.

[0089] For example, let's take the 3-state Kalman filter diagram shown in Figure 4A-2 as an example. In order from top to bottom, the first figure at the top is the brain signal waveform, the second figure is the 3-state command output diagram, the third figure is the 3-state Kalman filter output diagram, and the fourth figure is the 3-state filtered command output diagram. In the 3-state Kalman filter output diagram, the black line at the bottom is the output "threshold". Only when the filtered value exceeds the threshold will the system determine it as an SSVEP command, which greatly reduces the false trigger rate of the system and improves the stability of the system. This can be clearly seen from the fourth 3-state filtered command output diagram.

[0090] For example, let's take the 8-state Kalman filter diagram shown in Figure 4A-3 as an example. Following the order from top to bottom, the topmost figure is the output probability diagram of each command after filtering, with each spike representing a command; the second figure is the correlation coefficient diagram of each command. When the value of the first figure is 0, it represents the stage of no command output or resting state. The commands output at other times are the commands represented by the spike with the highest value at that time.

[0091] In this embodiment, the 8-state Kalman filter is applied inside the SSVEP to avoid cross-bonding within instructions during the instruction period. Its formula is as follows: x(t)=A1x(t-1)+A2[F(x(t), x(t-1))]+B

[0092] Here, the output probability x(t) of the instruction is an n×1 vector, where n is the total number of system instructions, t represents the current time, and t-1 represents the previous time. The value of each row of x(t) represents the probability of each SSVEP instruction being output at the current time. A1 and A2 are the system control matrices, responsible for controlling and coordinating the relationship between the previous instruction and the currently detected instruction, and for making outputs based on the actual situation. B is an n×1 constant vector used to make fine adjustments to the system based on the actual situation. F is the conditional function of the current instruction output probability x(t) based on the output probability x(t-1) of the previous instruction. F means that the current instruction output probability x(t) is calculated based on the output probability x(t-1) of the previous instruction and various actual situations.

[0093] For example, taking the schematic diagram shown in Figure 4A-4 as an example, if the diagram is taken during the user's resting period, the first image is the original classification diagram of the brain signal, and the second image is the Kalman filter diagram. A comparison of the two images shows that false triggers related to SSVEP have disappeared, maintaining a stable resting state. The false positive rate shown in Figure 4A-4 is 0%.

[0094] If Figure 4A-4 shows the user fixating on a stimulus block, it can be seen that the output instruction latency is very low and the consistency is very high. The data that can be given at this time are: the latency of the instruction entering after the left vertical line is 0, and the instruction execution rate is 100% throughout the fixation period.

[0095] In one embodiment of this application, as shown in FIG4B-1, step S410 of the embodiment of this application may further include steps S411 to S414.

[0096] S411, determine whether the target instruction identified in the current generation is the same as the target instruction identified in the previous generation;

[0097] S412, if the same, then the execution probability of the contemporary target instruction increases;

[0098] S413, if different, the execution probability of the contemporary target instruction decreases;

[0099] S414, update the execution probability of the target instruction to the execution probability of the current target instruction.

[0100] In this implementation, by adjusting the execution probability of the target instruction by either increasing or decreasing it, the accuracy and effectiveness of the target instruction output can be further improved. Individual increase / decrease adjustment strategies can be set for each target instruction, allowing for better matching with the actual control scenarios of external controlled devices. This enables flexible configuration based on the actual situation of the external controlled devices, ensuring the effectiveness of the brain-computer interface control device is unaffected by changes in the external controlled devices, resulting in a wide range of applications and greater versatility. For example, it can also achieve a fast increase and slow exit effect for a certain target instruction, which can be customized according to personal habits or system requirements.

[0101] In one embodiment of this application, as shown in Figure 4B-2, the curve on the left represents the process of increasing execution probability, and the curve on the right represents the process of decreasing execution probability. The difference lies in the selection of the "increasing" probability function and the "decreasing" probability function, respectively, where the delay in implementing the "increasing" and "decreasing" probability functions is controlled by the calculation parameters within the probability functions.

[0102] In one embodiment of this application, the accuracy of the execution and / or termination of the target instruction is controlled based on a probability function; the delay of the execution and / or termination of the target instruction is controlled based on the calculation parameters of the probability function; the probability function includes an increasing probability function and a decreasing probability function; the execution probability of the current target instruction is increased based on the increasing probability function; the execution probability of the current target instruction is decreased based on the decreasing probability function; each target instruction corresponds to its own probability function.

[0103] In one embodiment of this application, as shown in Figures 4C-1 and 4C-2, two schematic diagrams of target instruction execution modes are illustrated. This embodiment controls the "mode" of target instruction execution through a probability function. The probability function can be flexibly set as needed; it can be a curve mode, a straight line mode, or any other desired linear mode. This embodiment controls the "steepness" of the displayed curve of the probability function through calculation parameters. If the curve is particularly steep, the execution latency of the target instruction will be low (because the threshold will be reached quickly); if the curve is particularly flat, the execution latency of the target instruction will be high (because the threshold will be reached slowly).

[0104] Figure 4C-1 shows a schematic diagram of one mode of starting the execution of a target instruction. The black curve in the figure is the execution mode curve of the target instruction, indicating that the target instruction is executed smoothly with a slightly higher execution delay.

[0105] Figure 4C-2 shows another schematic diagram of the start of execution of the target instruction. The black curve in the figure is the execution mode curve of the target instruction, indicating that the target instruction is executed abruptly with low execution latency.

[0106] Different execution mode curves allow users to customize the system to suit their own needs, making the system more compatible with the habits of different users and achieving a balance between fewer accidental touches and faster response.

[0107] In this implementation, by setting the probability function and calculation parameters, the problem of "false triggering" in the target command output can be solved in practical applications. Taking game control as an example, false triggering refers to the unexpected operation performed by the game character at certain times. The source of this false triggering is that in reality, there will always be situations where one state is misjudged as another. To avoid this situation, this embodiment can implement a matching design between the target command output and the game control system based on a Kalman filter. This matching design considers the relationship between the currently output command and several previously executed commands before finally outputting the target command.

[0108] In one embodiment of this application, step S202 may involve feature recognition of the brain signal based on a response feature recognition model to obtain the brain response features contained in the brain signal. As shown in Figure 5, the control method of the brain-computer interface device described in this application embodiment may further include steps S501 to S503.

[0109] S501, Obtain a training dataset, which includes brain signals generated by the subject under various stimulus states;

[0110] S502, perform feature extraction on the training dataset (e.g., perform superimposed averaging on the same response feature data) to obtain a response feature template dataset;

[0111] S503, the response feature template dataset and the brain signal input response feature recognition model are processed to obtain the brain response features of the brain signal.

[0112] In one embodiment of this application, as shown in FIG6, the response feature recognition model 600 may include a first spatial filter 610, a correlation coefficient unit 620, and a response feature output unit 630.

[0113] The first spatial filter 610 performs filtering processing on the preset response feature template dataset to obtain the response feature filtered dataset.

[0114] The first spatial filter 610 filters the brain signal to obtain a first filtering feature.

[0115] The correlation coefficient unit 620 calculates the correlation value between the filtered feature and each response feature filtered data in the response feature filtered dataset, and determines the response feature corresponding to the response feature filtered data with the largest correlation value as the brain response feature of the brain signal.

[0116] The response feature output unit 630 outputs the brain response features.

[0117] This embodiment accurately identifies target response features in brain signals through the response feature recognition model, while suppressing non-target response features, thus providing a basic reference for achieving more accurate target instruction recognition in the future.

[0118] In one embodiment of this application, step S203 can perform instruction recognition on the brain signal based on the brain response features using a target instruction recognition model to obtain the target instruction represented by the brain signal. As shown in FIG7, the target instruction recognition model 700 includes a second spatial filter 710, a correlation processing unit 720, a classifier 730, and an instruction output unit 740.

[0119] The second spatial filter 710 filters the response feature template data corresponding to the brain response features to obtain the corresponding response feature filtered data.

[0120] The second spatial filter 710 filters the brain signal to obtain second filtered data.

[0121] The correlation processing unit 720 calculates and obtains the coefficient vector of the second filtered data and the response feature filtered data.

[0122] The classifier 730 classifies the coefficient vector and identifies the target instruction represented by the brain signal.

[0123] The instruction output unit 740 outputs the target instruction.

[0124] Based on the accurate identification of target response features in brain signals by the response feature recognition model, this embodiment further identifies target commands in brain signals more accurately through the target command recognition model, which greatly improves the information transmission rate of target commands in the control method of the brain-computer interface device.

[0125] In one embodiment of this application, the first spatial filter 610 and the second spatial filter 710 have the same structure and function, and are spatial filters of the same type. The process of obtaining the spatial filter includes: performing response feature analysis on the acquired training dataset to obtain the spatial filter; the response feature analysis includes task discriminant component analysis, task-related component analysis, or canonical correlation analysis; the training dataset includes brain signals generated by subjects under various stimulus states.

[0126] For example, Figure 8A of the embodiment provides a specific implementation diagram of online decoding and analysis of brain signals, including: the process before online decoding and analysis of brain signals and the process during online decoding and analysis of brain signals, which are described in detail below:

[0127] Before online decoding and analysis of brain signals, template data for each frequency is first generated based on the dataset collected in offline experiments (i.e., the training dataset). Then, a spatial filter w is trained using the Task-Discriminant Component Analysis (TDCA) algorithm. Finally, a linear discriminant analysis model corresponding to each frequency is generated by combining the spatial filter w and the training dataset.

[0128] In the online decoding and analysis process, the response characteristics of the brain signal under test are first identified, and then the target instruction of the brain signal under these response characteristics is identified. This allows for very accurate and rapid identification of the target instruction represented by the brain signal. The specific process includes:

[0129] 1) For a segment of test data (i.e., brain signals) to be tested, the correlation coefficient between the spatially filtered template data and the test data can be calculated to obtain the correlation with each frequency template. The larger the correlation coefficient, the stronger the correlation, indicating that the test data is highly likely to belong to the EEG response induced by the steady-state data stimulus of the instruction corresponding to the template data. By selecting the largest correlation coefficient, the encoding frequency of the target instruction can be determined.

[0130] 2) Target command identification is performed at each target frequency. This identification method involves extracting features from spatially filtered test data and target frequency template data, and then using a linear discriminant analysis model corresponding to the target frequency to identify and classify the extracted features. Five features were selected for feature extraction: the correlation coefficient between the spatially filtered template data and test data, the cross-correlation coefficient between the spatially filtered template data and test data, the frequency domain amplitude at the encoding frequency of the spatially filtered test data, the sum of the high-frequency domain amplitudes of the spatially filtered test data (including 55-72Hz), and the time domain variance of the spatially filtered test data. Among these, the correlation coefficient, cross-correlation coefficient, and frequency domain amplitude at the encoding frequency reflect the response characteristics of steady-state visual evoked potentials (SSVEPs) in brain signals, while the sum of the high-frequency domain amplitudes and the time domain variance reflect the response characteristics of teeth clenching in brain signals.

[0131] For example, taking the decoding analysis of 12 rounds of offline experimental data from a subject as an example, the recognition results are shown in Figure 8B, displaying the recognition results for 10 categories. The collected offline experimental data was divided into training and test datasets using leave-one-out cross-validation. In Figure 8B, labels 1-8 correspond to commands with frequencies of 8.5-12Hz (i.e., SSVEP commands), label 9 corresponds to rest commands, and label 10 corresponds to teeth-clenching commands. As shown in Figure 8B, the target command recognition accuracy (ACC) can reach 96.6%. When the induced SSVEP response is weak, commands at a certain encoding frequency are easily recognized as rest-encoded commands.

[0132] For example, taking the decoding and analysis of 12 rounds of offline experimental data from a subject as an example, the recognition results are shown in Figure 8C, displaying the recognition results for 8 categories. In Figure 8C, labels 1-8 correspond to instructions with frequencies of 8.5-12Hz, i.e., 8 SSVEP instructions. As can be seen from the results displayed in Figure 8C, the accuracy (ACC) of target instruction recognition can reach 95.8%.

[0133] Figures 8B and 8C show a confusion matrix, a classification model evaluation tool that visually illustrates the relationship between the model's predicted labels and the true labels. The confusion matrix can perform multi-class classification, visually displaying the model's performance by comparing the distribution of predicted and true labels. In multi-class problems, the confusion matrix becomes an n×n matrix, where n is the number of classes. Each row of the confusion matrix represents the actual class, and each column represents the predicted class. Each element (i, j) in the confusion matrix represents the number of samples that are actually of class i but are predicted as class j.

[0134] As shown in Figures 8B and 8C, the recognition accuracy of frequency-encoded commands in this embodiment is 95.8%, while the overall command recognition accuracy, including resting and teeth-clenching commands, is 96.6%. It can be seen that by including resting and teeth-clenching commands, this embodiment can increase the number of system commands while maintaining recognition accuracy, thereby increasing the system's degrees of freedom.

[0135] In one implementation of this application, an embodiment provides a brain-computer interface device, as shown in FIG9. The brain-computer interface device 900 includes an input interface module 910, a response feature recognition model 920, a target instruction recognition model 930, and an output interface module 940. The input interface module 910 acquires brain signals generated by a subject under any stimulus state; the response feature recognition model 920 performs response feature recognition on the brain signals to obtain the stimulus response features contained in the brain signals; the target instruction recognition model 930 performs instruction recognition on the brain signals under the response features to obtain the target instruction represented by the brain signals; and the output interface module 940 outputs the target instruction to control an external controlled device.

[0136] The modules / units described as separate components may or may not be physically separate. The components shown as modules / units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected to achieve the objectives of the embodiments of this application, depending on actual needs. For example, the functional modules / units in the various embodiments of this application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0137] In one implementation of this application, an embodiment provides an electronic device, as shown in FIG10. The electronic device includes one or more processors and one or more memories. The memories store computer-readable code, which, when executed by the one or more processors, implements the control method of the brain-computer interface device as described above. Furthermore, the electronic device may also include conventional electronic components such as I / O interfaces and communication modules, which will not be elaborated upon here.

[0138] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0139] In one implementation of this application, an embodiment of this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the control method of the brain-computer interface device as described above.

[0140] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing a processor. The program can be stored in a computer-readable storage medium, which is a non-transitory medium, such as random access memory, read-only memory, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof. The storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. This available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state drive (SSD)).

[0141] In one implementation of this application, an embodiment may also provide a computer program product comprising one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in the embodiments of this application are generated. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. When the computer program product is executed by a computer, the computer performs the methods described in the foregoing method embodiments. The computer program product may be a software installation package; when the foregoing methods are required, the computer program product may be downloaded and executed on a computer.

[0142] In the embodiments provided in this application, it should be understood that the disclosed systems, devices, or methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, modules, or units, and may be electrical, mechanical, or other forms.

[0143] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0144] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A control method for a brain-computer interface device, characterized in that, include: Acquire brain signals generated by subjects under any stimulus state; The brain signals are subjected to feature recognition to obtain the brain response features contained in the brain signals; The brain signal is subjected to instruction recognition under the brain response features to obtain the target instruction represented by the brain signal; Based on the target instruction, a preset execution probability is configured, and based on the execution probability, the target instruction is output to control an external controlled device.

2. The control method for the brain-computer interface device according to claim 1, characterized in that, Also includes: The execution probability of the identified target instruction is calculated iteratively based on the Kalman filter; each time the target instruction is identified, the corresponding iterative calculation increases by one generation. Determine whether the execution probability of the target instruction is greater than or equal to a preset value; If so, the target instruction is output to control the external controlled device; If not, the target instruction will not be output.

3. The control method for the brain-computer interface device according to claim 1, characterized in that, Also includes: Determine whether the target instruction identified in the current generation is the same as the target instruction identified in the previous generation; If they are the same, the probability of executing the contemporary target instruction increases; If they are different, the execution probability of the contemporary target instruction decreases; The execution probability of the target instruction is updated to the execution probability of the current target instruction.

4. The control method for the brain-computer interface device according to claim 3, characterized in that, Also includes: The accuracy of the execution and / or termination of the target instruction is controlled based on a probability function; The delay in the execution and / or termination of the target instruction is controlled based on the calculation parameters of the probability function. The probability function includes an increasing probability function and a decreasing probability function; the increasing probability function controls the execution probability of the current target instruction to increase; the decreasing probability function controls the execution probability of the current target instruction to decrease; each target instruction has its own probability function.

5. The control method for the brain-computer interface device according to claim 1, characterized in that, Also includes: The brain response features include visual brain response features and non-visual brain response features; the non-visual brain response features include muscle activity brain response features and resting state brain response features. The target instruction recognition model identifies the brain signal based on the visual brain response features to obtain the operation-type instructions and / or movement-type instructions represented by the brain signal; The target instruction recognition model identifies the brain signal based on the muscle activity brain response characteristics to obtain the movement-type instructions and / or control-type instructions represented by the brain signal; The target instruction recognition model identifies the brain signal based on the resting-state brain response characteristics to obtain the resting-state instruction represented by the brain signal; The resting state instruction refers to an instruction that does not execute the operation instruction, the movement instruction, and the control instruction; the control instruction enables the starting, stopping, or / and switching of the execution of other instruction types.

6. The control method for the brain-computer interface device according to claim 5, characterized in that, The resting state instruction also includes retaining the original action of the previous instruction when the operation instruction, the movement instruction, and the control instruction are not executed.

7. The control method for the brain-computer interface device according to claim 5, characterized in that, Also includes: The subject's visual orientation signal is acquired using an inertial measurement unit; The viewpoint direction signal is used for instruction recognition to obtain that the target instruction represented by the viewpoint direction signal is a viewpoint instruction and / or a direction instruction; the viewpoint instruction and / or direction instruction can be executed simultaneously with the operation instruction, movement instruction, control instruction and / or resting state instruction.

8. The control method for the brain-computer interface device according to claim 1, characterized in that, Also includes: Based on a response feature recognition model, the brain signal is subjected to feature recognition to obtain the brain response features contained in the brain signal, including: Acquire a training dataset, which includes brain signals generated by subjects under various stimulus states; Feature extraction is performed on the training dataset to obtain a response feature template dataset; The brain response features of the brain signal are obtained by processing the response feature template dataset and the brain signal input response feature recognition model.

9. The control method for the brain-computer interface device according to claim 8, characterized in that, The response feature recognition model includes a first spatial filter, a correlation coefficient unit, and a response feature output unit; The first spatial filter performs filtering on the preset response feature template dataset to obtain the response feature filtered dataset; The first spatial filter filters the brain signal to obtain a first filtering feature; The correlation coefficient unit calculates the correlation value between the filter feature and each response feature filter data in the response feature filter dataset, and determines the response feature corresponding to the response feature filter data with the largest correlation value as the brain response feature of the brain signal. The response feature output unit outputs the brain response features.

10. The control method for the brain-computer interface device according to claim 9, characterized in that, Also includes: Based on a target instruction recognition model, the brain signal is used to identify the instruction under the brain response features to obtain the target instruction represented by the brain signal. The target instruction recognition model includes a second spatial filter, a correlation processing unit, a classifier, and an instruction output unit; The second spatial filter filters the response feature template data corresponding to the brain response features to obtain the corresponding response feature filtered data; The second spatial filter filters the brain signal to obtain second filtered data; The relevant processing unit calculates and obtains the coefficient vector of the second filtered data and the response feature filtered data; The classifier classifies the coefficient vector and identifies the target instruction represented by the brain signal; The instruction output unit outputs the target instruction.

11. The control method for the brain-computer interface device according to claim 9 or 10, characterized in that, Also includes: Response feature analysis is performed on the acquired training dataset to obtain spatial filters; The response feature analysis includes task discriminant component analysis, task-related component analysis, or canonical correlation analysis.

12. A brain-computer interface device, characterized in that, include: The input interface module acquires brain signals generated by the subject under any stimulus state. A response feature recognition model is used to identify the response features of the brain signals to obtain the stimulus response features contained in the brain signals. A target instruction recognition model performs instruction recognition on the brain signal under the response features to obtain the target instruction represented by the brain signal; The output interface module outputs the target command to control the external controlled device.

13. An electronic device, characterized in that, include: One or more processors; and One or more memories, wherein computer-readable code is stored in the memories, the computer-readable code, when executed by the one or more processors, implements the control method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the control method as described in any one of claims 1 to 11.

Images