Method of processing brain signals

Abstract

The invention relates to methods for detecting brain states by reading bioelectrical data. To determine a similarity between a person's brain states at different times, received signals are assessed for the presence or absence of an information component in common with signals corresponding to specific brain states. Brain signals are received by means of brain-computer interface sensors arranged on the head. The received signals are digitized, and the digitized signal of each sensor is converted into a topological space object. The converted signals are assessed to establish whether they belong to a pre-constructed topological space for a specific brain state, wherein if a converted signal belongs to such a topological space, said signal has an information component in common with the signal for the corresponding brain state, and the state of the brain under examination is similar to said state. The technical result is that of enabling faster and more informative detection of a common information component while maintaining high accuracy, allowing real-time assessment of the similarity of a state of a brain under examination with a specific state in a database of states.

Description

[0001] METHOD OF PROCESSING BRAIN SIGNALS

[0002] AREA OF TECHNOLOGY

[0003] The invention relates to the field of detecting brain states, in particular to equipment and methods for detecting brain states by reading its bioelectrical data.

[0004] LEVEL OF TECHNOLOGY

[0005] A brain-computer interface (BCI) is known from prior art (Russian Patent No. 2823580) and includes a specialized input device for obtaining electromagnetic or other information about the brain. The BCI includes an analog-to-digital converter, microcontrollers for processing information, and may contain software and hardware data converters, such as modules for obtaining frequency spectra and spatial and temporal domain data in the brain signal, a GPU / TPU / CPU / neuromorphic processor for training and operating a classifier, which may be integrated into the BCI or located separately, including on a training server, a hardware module for storing and operating the trained / trained / customized classifier, and a neural network.In this case, neural networks are used to process the received brain signals. An additional computing device containing a GPU / TPU / CPU / neuromorphic processor, or a training server, is used for training and operation. Incorporating these additional modules into the interface device significantly increases energy consumption and signal processing time, thereby preventing real-time processing and classification of brain signals. This system configuration also prevents the entire neural interface from being placed on the user's head, thereby ensuring wearable mobility and autonomy. Therefore, the disadvantage of this solution is the use of a graphics processor (or other additional computing devices) for its operation and, consequently, the low data recognition speed, which precludes real-time signal processing. Furthermore, the interface in question suffers from low energy efficiency, limited mobility, and limited autonomy.

[0006] The proposed technical solution is aimed at eliminating the shortcomings of the current state of technology and differs from previously known ones in that it provides higher signal processing speed, autonomy, mobility, and increased energy efficiency while maintaining or increasing the accuracy of brain pattern recognition, compared to other state-of-the-art solutions.

[0007] DISCLOSURE OF THE INVENTION

[0008] The technical problem solved by the claimed invention is to increase the speed and information content of detecting the presence or absence in brain signals of an information component characteristic of a certain state of the brain of a given individual or group of individuals while maintaining high accuracy.

[0009] The technical result of the claimed invention is to increase the speed and information content of detecting the presence or absence in brain signals of an information component characteristic of a certain state of the brain of a given individual or group of individuals while maintaining high accuracy, making it possible to evaluate in real time the similarity of the state of the brain being studied with a certain state from the state database.

[0010] The claimed technical result is achieved by determining the similarity between the states of the brain of one person at different points in time or between different people by determining the presence or absence in the received signals of a common information component with signals corresponding to a specific state of the brain; by receiving brain signals through sensors of a neurointerface located on the head; by digitizing the received signals; by converting the digitized brain signal from each sensor into an object of topological space;the belonging of the transformed brain signals to a topological space, previously constructed for a certain state of the brain, is assessed, and if the transformed brain signal belongs to such a topological space, then the brain signal has a common information component with the signal for the corresponding state of the brain, and if the brain signal or signals have a common information component with the signal for the corresponding state of the brain, then the state of the brain being studied is similar to such a state. IMPLEMENTATION OF THE INVENTION;

[0011] The following detailed description of the invention includes numerous implementation details intended to provide a clear understanding of the present invention. However, one skilled in the art will readily appreciate how the present invention may be utilized with or without these implementation details. In other instances, well-known methods, procedures, and components have not been described in detail to avoid unnecessarily obscuring the features of the present invention.

[0012] Furthermore, it will be clear from the foregoing description that the invention is not limited to the embodiment described. Numerous possible modifications, changes, variations, and substitutions, while preserving the spirit and form of the present invention, will be apparent to those skilled in the art.

[0013] The proposed method is implemented using a hardware and software system for detecting the presence of a common information component in signals received from at least one electromagnetic signal source in the brain. The data can be presented as time series obtained from electromagnetic, chemical, or acoustic signals.

[0014] The system at least comprises a neural interface (input device). The neural interface (input device) is a hardware and software complex and comprises a data acquisition unit with 8 (or 16, or 32, a countable number) gold-plated electrodes with a spring mechanism, arranged according to the international 10-10 / 10-20 schemes. The neural interface contains a power source, which can be a battery or other means. The neural interface also contains means for converting an analog signal into digital form, for example, the neural interface may contain 2 data digitization units with microcontrollers with an ADS1299 analog-to-digital converter (ADC). The neural interface may contain data packaging means, for example, 3 data packaging units according to the author's protocol, including adding to the time series of digitized data marks about the beginning and end of a packet, a hash function, a numerical value of the length of the time series, service information about the device.The neural interface also has a classification unit implemented on programmable logic integrated circuits, such as the Xilinx family, or on microcontrollers, such as the ESP32.

[0015] To transmit data from the neural interface, a data transmission unit / units are used via a wireless connection, such as Bluetooth. One implementation of the present invention may utilize a data transmission unit using an ESP microcontroller with an integrated BLE 5.2 module. The neural interface may also include a memory unit for storing a database of states. Data obtained in the classification unit can be sent to a mobile computer via the BLE protocol for biofeedback and / or for visualization and further analysis of the classified data.

[0016] The system may additionally include a mobile computer (smartphone) and a desktop computer. The mobile computer's specifications may include: Android version 12 (>12 recommended), Android API level >31, NPU support, 4 GB of RAM (8 GB recommended), and a screen >7 inches. The mobile computer receives data from the classification unit via the BLE protocol. The mobile computer can also receive a vector of values ​​and further classify the data using a neural network in inference mode and transmit the data via a gigabit Wi-Fi protocol to the desktop computer.

[0017] The desktop computer can be equipped with a graphics processing unit (GPU) and a central processing unit (CPU). The desktop computer has data storage with a RAID 5 data loss protection unit. The computer performs data transformation and trains the neural network using data received from the neural interface via a mobile computer or directly via a wireless or wired connection. As mentioned earlier, data classification using a neural network is an additional tool. Data classification using a neural network can serve as the primary classification tool when the patented method is used to represent neural interface data as data with topological features. The neural network architectures on the desktop computer include fully convolutional, convolutional, recurrent neural networks, generative networks, transformers, and spike networks.The stationary and mobile computers operate in interactive mode. The stationary computer transmits the modified network architecture and / or network parameters after training to the mobile computer in inference mode.

[0018] In humans, 2-25 electromagnetic signal generators (EMGs) can be identified in the brain. These generators are located in the cortical and subcortical structures of the brain. These generators represent a morphofunctional agglomeration of neurons that simultaneously perform a common brain function, which is represented by the generation of a quasi-periodic signal. The number and arrangement of EMGs varies depending on the brain's state and is individualized. When the brain is in a certain state at a given moment in time, a certain number of oscillators (generators) of quasi-periodic electromagnetic signals are active. Each oscillator is represented by a sum of smaller oscillators (neurons). The signals received from each sensor of the neural interface represent a combination of signals from active oscillators in the brain. Thus, brain states are characterized by a combination of oscillator activity.This representation of brain signals as a sum of signals from nearly periodic signal generators makes it possible to describe the brain signal as a set of harmonic signals described by nearly periodic functions.

[0019] Thus, brain states can be described by a system of differential equations with constant coefficients for each sensor of the following form: y(t)

[0020]

[0021] = En a n (t)e lnt sm(co n t + <р п ), where a n (t) - polynomials, N is the number of oscillators.

[0022] Consequently, in similar states of the brain of people or of one person at different moments in time (for example, emotions of joy, states of concentration, a meditative state, a state of emotional uplift), a common information component is revealed between brain signals represented by a certain number of brain oscillators and their characteristics (location, power).

[0023] The proposed invention proposes to evaluate the similarity of the state of the brain based on the nature of brain signals in response to certain external and internal stimuli by identifying common information components in brain signals.

[0024] Moreover, this similarity of brain states can be detected for either one or several individuals. If the similarity of a brain state is assessed for a single individual, the brain state is compared with the database of brain states collected from that individual. If the similarity of a brain state is assessed for a group of people, the brain state is compared with the set of brain states of that group of people. Moreover, the similarity of states is assessed by detecting the presence or absence of a common information component in the signals received by the sensors with the recorded signals corresponding to a specific brain state in the database of brain states.

[0025] The analog brain signals read by the electrodes (sensors) of the neurointerface have a range of 5-100 microvolts (µV). According to biophysical data, the frequency spectrum of the physiological electromagnetic brain signal lies in the range of 1-100 Hz. A frequency of 250 Hz is sufficient for signal analysis. The ADC converts the analog signal with a sampling frequency of 250 Hz. One measurement is packed into a 32-bit vector, of which 26 bits are informative and 6 bits are service. The signal error is 3 bits. Analysis of real data shows that all 26 bits are used significantly. The vector of values ​​is packed every 100 milliseconds (ms) in the data packing block and sent to the classification block. In the classification block, the similarity between the brain signals read by the sensors and the brain signals for a specific state in the state database is assessed by identifying common information components (invariants).To do this, the digitized physiological electromagnetic signal of the brain is represented as an object of topological space and the topological properties of the resulting time series scans and the number of signal generators (oscillators) are evaluated, having previously configured the classifier.

[0026] As mentioned earlier, to identify a common topological invariant and the number of signal sources, the brain signal (time series) from each sensor is transformed into an object in a topological space and analyzed in that space. The analysis is based on the transition from a time series to a multidimensional unfolding—a piecewise linear curve in a multidimensional space whose points correspond to segments of the original series—and an analysis of the projections of this curve onto subspaces spanned by the principal components (vectors) of the corresponding scattering matrix.

[0027] A time series f = (fi, fi) is a sequence of values ​​f(t), with a fixed time interval At.

[0028] Each digitized physiological electromagnetic signal from the brain, received as a time series (digitized sequence) from a sensor or sensors, must be assigned to one of m equivalence classes. Each equivalence class corresponds to a brain state being compared. This requires transforming the brain signal using the method described below and assessing whether the transformed signal fits into a topological space previously constructed for the brain state being compared. This is accomplished by performing steps S1–S3.

[0029] S1 Receive a digitalized discrete brain signal in the form of real-valued time series f 1 = (f 1 i, ..., Gm), ..., f s = (f s i, ..., f s N) of fixed length N, where s is the number of neural interface sensors.

[0030] Next, a multidimensional (n-dimensional) scan is obtained from each time series.

[0031] If there is no a priori information about the expected number of signal sources, the maximum possible scan window n is selected. The maximum possible initial value n = Q((N + 1) / 2), where Q is the integer part. This results in a Hankel matrix of size n x p, where p = N - n + 1, with rank < n.

[0032] In this case, the n-dimensional scan Xf = Xf(N, n) of the time series f = (fi, ..., fi\i) in the n-dimensional Euclidean space is an oriented piecewise linear curve obtained by successively connecting the vectors Xi, X2, ..., Xp, where Xi = (fi, fi+1 , ..., f i+nl).

[0033] From s time series (obtained from s sensors) we obtain time series scans in the form of Xf curves (namely, Xn .. Xf curves) S — taking into account the index with the sensor number) with nodes that are described by the Hankel matrix.

[0034] Thus, the piecewise linear curve Xf of the sweep of each time series of the corresponding sensor can be written in the form of a matrix, where the columns of the matrix are n-dimensional vectors or nodes of the curve Xf: X1 2, .... X р , and the resulting matrix of p columns and n rows is a Hankel matrix of the form: "A A / 3 ■" fN-n+l

[0035] A / 3 A ■" A / — n+2

[0036] A A A "■ fN-n+3

[0037]

[0038] -fn fn+1 fn+2 "■ A /

[0039] Next, scattering matrices W are formed (one matrix for each sensor) of the Xf curves (obtained from the sensors), and eigenvalues ​​and eigenvectors are obtained for the scans from each channel (sensor) of the neural interface data.

[0040] The rank r of the scattering matrix W of the Xf curves is calculated as the maximum number of eigenvalues ​​such that the remaining eigenvalues ​​can be neglected, taking into account the noise criterion.

[0041] In this case, the rank r (or E-rank r, when the eigenvalues ​​fall in the E-neighborhood) corresponds to the number of active signal sources - oscillators as follows: q = r / 2, q is the number of active signal sources (oscillators) that generate the signal arriving at a certain sensor.

[0042] 52 Project the Xf curves onto the subspaces spanned by the first r principal components (eigenvectors whose eigenvalues ​​are greater than the value estimated as noise). This results in Xf curves in r-dimensional space. This eliminates noise.

[0043] The Xf curves are projected onto the subspaces spanned by the principal components (vectors) of the corresponding scattering matrix. The resulting set of projections is fed to the classifier for classifying the neural interface data for brain state assessment.

[0044] 54 Carry out classification or clustering.

[0045] S4_1 At this step, the following proposed classifier can be used. Equivalence classes of time series are formed and a subspace is constructed for m equivalence classes {Ki,..., Km}, corresponding to m brain states. This step is performed on brain signal data for which the brain state is known. Each class corresponds to its own hyperplane with an s-neighborhood for each sensor of the neural interface. E is selected such that the resulting hyperplanes do not lie in each other's E-neighborhoods. The following algorithm can be used to implement the above:

[0046] The algorithm is fed with s time series as input.

[0047]

[0048] fs}, fi = (f'-i,..., Km), grouped into the corresponding equivalence classes {Ki , ... , Km}. Further:

[0049] (1) We obtain the unfolding of s time series.

[0050] (2) We obtain the ranks of the piecewise linear curves, and the parameter r—the subspace dimension—corresponds to the rank (described in S1). We obtain r corresponding to the criterion—the maximum possible, taking into account the noise criterion, or the value necessary and sufficient for a given accuracy of brain state classification. Moreover, r can be different for each sensor.

[0051] (3) In the resulting subspace of dimension r, we construct m pieces of r-dimensional hyperplanes {Li,..., L m} with E-neighborhoods for each equivalence class {Ki,..., Km}. We assign a value Ei to each hyperplane such that the scans of one class lie in the Ei-neighborhood of the corresponding hyperplane Li, expressed, for example, by the corresponding confidence interval (1a, 2a, 3a). We obtain: {EI,..., Em} are sets for m equivalence classes (and m x g sets for g sensors).

[0052] (4) We check whether {Li, ..., Lm} lie in £-neighborhoods of each other.

[0053] for L, Li e {Li , ..., Lm} do

[0054] for Lj, Lj e {Li, ..., Lm}, ji do

[0055] if Lj Ei is a neighborhood of Li then

[0056] g t <-

[0057]

[0058] Other approaches are possible. For example, (a) pointwise removal from Li or (b) removal from Li of the corresponding unfolding and recalculation of Li and Ei.

[0059] end if

[0060] end for

[0061] end for

[0062] We get: updated values ​​{EI, ..., Em}.

[0063] Output: {Li ,..., Lm}, {ei , ..., £m}.

[0064] The subspaces of hyperplanes {Li,..., Lm} with E-O crosses obtained for the equivalence classes of brain states allow the formation of a classifier for assessing the membership of a brain signal in an equivalence class to the corresponding brain state. The accuracy of this classifier can be optimized using supervised learning. The membership of the studied brain signal from the neural interface sensors is assessed based on the membership of the scan of the corresponding time series.

[0065] £-neighborhoods of r-dimensional hyperplanes corresponding to a certain class of brain states.

[0066] The similarity assessment of the signal under study and the selected signal in a certain state from the database (or recorded in real time from another individual) is carried out by classification.

[0067] The brain signal under study, represented as an n-dimensional scan Xf = Xf(N, n) of a time series f, belongs to the £-neighborhood of the r-dimensional hyperplane Li if E

[0068]

[0069] f=i(||^i -Х / ||) 2 < £. If the signal under study belongs to the £-neighborhood of the corresponding class, then the brain signal under study is similar to brain signals from this class.

[0070] To evaluate the similarity of the signal under study, the algorithm below can be used, the input of which is fed with a time series fi = (f'i,..., PN) from a set of time series {f1,..., fs}, hyperplanes {Li, ..., Lm}, {EI, ..., £ т}-values, {Ki, ..., Km}, obtained during the construction of the classifier (described above).

[0071] (1) We apply the multidimensional sweep method to the time series fi= (f'i ,..., PN). We obtain an n-dimensional sweep Xf .

[0072] (2) We check whether the development Xf (piecewise linear curve) falls into the E-neighborhood of the corresponding hyperplane from {Li, ..., Lm} of m classes:

[0073] for Li e {Li , .... Lm} do

[0074] if E

[0075]

[0076] f =1 (||bf - X / ||) 2 < £ then

[0077] fi e (3) The output from the algorithm is the value i, which corresponds to the class of brain states.

[0078] end if

[0079] end for

[0080]

[0081] if - X^||) 2 > £ for any Li from {Li, ..., Lm} then

[0082] (4) Output: i corresponding to Li with minimum distance X and Li from {Li, ..., Lm}

[0083] end if

[0084] Thus, at the output, we assign each digitized brain signal to one of the equivalence classes. Signals belonging to the same class share a common information component. If a brain signal or signals share a common information component with the signal for the corresponding brain state from the brain state database, then the state of the brain being studied is similar to that state. Moreover, the conditions for state similarity may vary and be established depending on the state being determined. For example, a condition may be established under which if at least one brain signal read by the sensors shares a common component with the signal for the corresponding brain state from the brain state database, then the state of the brain being studied is similar to that state. There may be cases where multiple signals with a common component are required to recognize state similarity.Or, in cases where brain signals generated by specific signal sources and detected by corresponding sensors are more significant for determining state similarity, then recognizing state similarity requires the presence of a common component in such signals (or a set of such signals). To improve the accuracy of brain state recognition and the detection of a common component, the method can be supplemented with neural network classifiers.

[0085] At this step, the classifier can be represented by a neural network whose input is the set of projections from the previous step, S3, packaged into a multidimensional input vector. The classifier can be implemented using either supervised learning (with a labeled data sample indicating the corresponding brain state) or unsupervised learning (with prior information about the presence of brain state clusters). Thus, at the output, each desired digitized brain signal is assigned to one of the equivalence classes with a specified accuracy.

[0086] At this step, a clustering algorithm can be used that implements “unsupervised learning” and allows, having received as input a set of projections from step S3, to partition the data into clusters according to the common information component identified in step S1-S3.

[0087] Consequently, this method enables one to correlate a subject's brain state with the brain states of other subjects or the same subject at a different time by searching for common information components. Moreover, unlike other signal analysis technologies (fully convolutional, convolutional, recurrent neural networks, generative networks, transformers, and spiking networks), the method can be implemented on programmable gate arrays (FPGAs) and application-specific integrated circuits (ASICs) in the form of efficient matrix calculations, with an accuracy no worse than the required (or worse than other technologies). It also avoids the energy-intensive training associated with other technologies (e.g., neural networks) and the use of additional devices (in most cases, a graphics processing unit (GPU)) for such training and operation, which also significantly increases energy consumption.Consequently, FPGAs and specialized integrated circuits can be located within the neural interface itself, significantly increasing the speed of signal processing and common component identification, as classification is performed within the neural interface itself, without remotely accessing the server. This enables real-time recognition of brain states while maintaining high accuracy. This has enabled us to increase the speed of movement recognition (for example, a signal to control the right or left hand) to 10 movements per second, significantly higher than with other technologies.

[0088] The proposed solution also improves the accuracy of mental movement recognition—from 0.05 for 30 mental movements to 0.8 for 30 mental movements.

[0089] The proposed solution also improves the energy efficiency of the neural interface, ensuring its mobility and autonomy.

[0090] Although the invention has been described with reference to the disclosed embodiments, it will be apparent to those skilled in the art that the specific experiments described in detail are provided merely for the purpose of illustrating the present invention and should not be construed as limiting the scope of the invention in any way. It should be understood that various modifications are possible without departing from the spirit of the present invention.

Claims

Formula 1. A method for processing brain signals to determine the similarity between the brain states of one person at different points in time or of different people by detecting the presence or absence in the received signals of a common information component with signals corresponding to a specific brain state, comprising the steps of: receive brain signals through neurointerface sensors placed on the head; carry out digitalization of received signals; transform the digital brain signal from each sensor into an object of topological space; The belonging of the transformed brain signals to a topological space, previously constructed for a certain state of the brain, is assessed, and if the transformed brain signal belongs to such a topological space, then the brain signal has a common information component with the signal for the corresponding state of the brain, and if the brain signal or signals have a common information component with the signal for the corresponding state of the brain, then the state of the brain being studied is similar to such a state.

2. The method according to claim 1, wherein the sensors are electrodes and are designed to record an electroencephalogram of the human brain.

3. The method according to claim 1, wherein, when converting a digitized brain signal into an object of a topological space, the time series of the digitized signal is converted into a piecewise linear curve that is representable by a Hankel matrix; a scattering matrix is ​​obtained, sets of eigenvalues ​​and sets of eigenvectors are obtained; the rank of such a piecewise linear curve is calculated, wherein the rank of the piecewise linear curve linearly corresponds to the number of signal sources that together produce a signal on the sensor.

4. The method according to claim 1, wherein the brain signals are represented as a set of harmonic signals described by almost periodic functions.

5. The method according to claim 1, wherein the information about brain states is stored in a brain state database.

Images