Method of activating a stored brain pattern

Abstract

The invention relates to the field of detecting brain states and activating a stored brain pattern. Brain activity is recorded while a particular state is being experienced in order to obtain a brain activity pattern for a particular brain state. Recordings of brain activity patterns for different brain states are stored in a library of mental states on an electronic medium. A user's current brain state and the corresponding brain activity pattern are obtained using a brain-computer interface. If the brain state does not match a desired pattern, a stored brain pattern is activated using feedback, while the distance between the brain state and the desired pattern is monitored. The result is that of increasing the speed and accuracy with which a user's current brain state is identified, while maintaining a high degree of accuracy, and also making it possible to store a brain state in the form of a brain activity pattern and also to activate a desired brain pattern from a library of mental states.

Description

[0001] A METHOD FOR ACTIVATING A STORED BRAIN PATTERN

[0002] AREA OF TECHNOLOGY

[0003] The invention relates to the field of detecting brain states and methods for activating a stored brain pattern, in particular to equipment and methods for detecting brain states by reading its bioelectrical data.

[0004] LEVEL OF TECHNOLOGY

[0005] From the prior art, Russian Federation Patent No. 2823580, a brain-computer interface is known, containing a specialized input device for receiving electromagnetic or other information about the brain and training the user to engage in mental control of identified areas of the brain with degenerative changes or selected areas of the brain to improve intellectual functions and / or acquire new intellectual abilities and skills when implementing mental control in a virtual environment or devices using a brain-computer interface. The brain-computer interface includes an analog-to-digital converter, microcontrollers for processing information, may contain software and hardware information converters, for example, in the form of modules for obtaining a frequency spectrum, spatial and temporal domain data in the brain signal, GPU / TPU / CPU / neuromorphic processor for training and operating the classifier,This can be integrated into the brain-computer interface or located separately, including a hardware module for storing and operating the trained / trained / customized classifier and neural network on the training server. In this case, neural networks are used to process the received brain signals, and an additional computing device containing a GPU / TPU / CPU / neuromorphic processor, or a training server, is used for training and operation. Incorporating these additional modules into the interface device significantly increases energy consumption and signal processing time, thereby preventing the processing and classification of brain signals in real time. This system configuration also prevents the entire neural interface from being placed on a person's head, ensuring wearable mobility and autonomy. This system also does not provide the ability to change the brain state to the desired one. Thus, the disadvantage of this solution isThe use of a graphics processor (or other additional computing devices) for its operation and, consequently, low data recognition speed, which precludes real-time signal processing, which prevents the interface from changing to the desired state. Also, the interface in question has low energy efficiency, low mobility, and limited autonomy.

[0006] The proposed technical solution is aimed at eliminating the shortcomings of the current state of technology and differs from previously known ones in that it provides higher signal processing speed, autonomy, mobility, and increased energy efficiency while maintaining or increasing the accuracy of brain pattern recognition, compared to other state-of-the-art solutions.

[0007] DISCLOSURE OF THE INVENTION

[0008] The technical problem solved by the claimed invention is to increase the speed and accuracy of detecting the current state of the brain, to ensure the ability to save the state of the brain in the form of a pattern of brain activity and to activate the required brain pattern in the user of the neurointerface using feedback.

[0009] The technical result of the claimed invention is to increase the speed and accuracy of detecting the current state of the user's brain while maintaining high accuracy, ensuring the ability to save the state of the brain in the form of a pattern of brain activity and ensuring the ability to activate the required brain pattern from a library of mental states.

[0010] The claimed technical result is achieved by recording brain activity during the experience of a certain state, obtaining a pattern of brain activity for a certain brain state; storing recordings of patterns of brain activity for various brain states in a library of mental states on an electronic medium; when the user sets the desired brain state from the library of mental states, loading the pattern of the corresponding state in the form of weights of the classifier of electromagnetic signals of the user's brain; receiving brain signals from the user via sensors of the neurointerface located on the user's head to determine the current state of the user's brain and obtaining the corresponding pattern of brain activity;compare the current state of the brain, represented by the corresponding partner of brain activity, with the required state, represented by the corresponding partner of brain activity, and in the event of a discrepancy between the states, activate the stored brain pattern using feedback, wherein the feedback is implemented by means of an indicator that displays the distance of the current state, represented by the corresponding partner of brain activity, from the required state, represented by the corresponding partner of brain activity.

[0011] IMPLEMENTATION OF THE INVENTION

[0012] The following detailed description of the invention includes numerous implementation details intended to provide a clear understanding of the present invention. However, one skilled in the art will readily appreciate how the present invention may be utilized with or without these implementation details. In other instances, well-known methods, procedures, and components have not been described in detail to avoid unnecessarily obscuring the features of the present invention.

[0013] Furthermore, it will be clear from the foregoing description that the invention is not limited to the embodiment described. Numerous possible modifications, changes, variations, and substitutions, while preserving the spirit and form of the present invention, will be apparent to those skilled in the art.

[0014] The proposed method is implemented using a hardware and software system for detecting differences between stored signals and signals coming from at least one electromagnetic signal source in the brain, as well as feedback mechanisms that enable activation of the desired brain patterns. The data can be presented as time series obtained from the electromagnetic signal.

[0015] The system at least comprises a neural interface (input device). The neural interface (input device) is a hardware and software complex and comprises a data acquisition unit with 8 (or 16, or 32, a countable number) gold-plated electrodes with a spring mechanism, arranged according to the international 10-10 / 10-20 schemes. The neural interface contains a power source, which can be a battery or other means. The neural interface also contains means for converting an analog signal into digital form, for example, the neural interface may contain 2 data digitization units with microcontrollers with an ADS1299 analog-to-digital converter (ADC). The neural interface may contain data packaging means, for example, 3 data packaging units according to the author's protocol, including adding to the time series of digitized data marks about the beginning and end of a packet, a hash function, a numerical value of the length of the time series, service information about the device.The neural interface may also contain a magnetometer.

[0016] The neural interface also has a classification unit implemented on programmable logic integrated circuits, such as the Xilinx family, or on microcontrollers, such as the ESP32.

[0017] To transmit data from the neural interface, a data transmission unit / units are used via wireless communication, such as Bluetooth and Wi-Fi. One implementation of the present invention may utilize a data transmission unit using an ESP microcontroller with an integrated BLE 5.2 module. The neural interface may also include a memory unit for storing a state database. Data obtained in the distance calculation unit can be sent to a mobile computer via the BLE protocol for biofeedback and / or for visualization and further analysis of the classified data.

[0018] The system may additionally include a mobile computer (smartphone) and a desktop computer. The mobile computer's specifications may include: Android version 12 (>= 12 recommended), Android API level >= 31, NPU support, 4 GB of RAM (8 GB recommended), and a screen >= 7 inches. The mobile computer receives data from the classification unit via the BLE protocol. The mobile computer can also receive a vector of values ​​and further classify the data using a neural network in inference mode and transmit the data via a gigabit Wi-Fi protocol to the desktop computer.

[0019] The desktop computer can be equipped with a graphics processing unit (GPU) and a central processing unit (CPU). The desktop computer has data storage with a RAID 5 data loss protection unit. The computer performs data transformation and trains the neural network using data received from the neural interface via a mobile computer or directly via a wireless or wired connection. As mentioned earlier, data classification using a neural network is an additional tool. Data classification using a neural network can serve as the primary classification tool when the patented method is used to represent neural interface data as data with topological features. The neural network architecture on the desktop computer includes the following types: fully convolutional, convolutional, recurrent neural networks, generative networks, transformers, and spike networks.The stationary and mobile computers operate in interactive mode. The stationary computer transmits the modified network architecture and / or network parameters after training to the mobile computer in inference mode.

[0020] In humans, 1-25 electromagnetic signal generators (EMGs) can be identified in the brain. These generators are located in the cortical and subcortical structures of the brain. These generators represent a morphofunctional agglomeration of neurons that simultaneously perform a common brain function, which is represented by the generation of a quasi-periodic signal. The number, nature, and location of these EMGs vary depending on the brain's state and are individualized. When the brain is in a certain state at a given moment in time, a certain number of oscillators (EMGs) are active. Each oscillator is represented by a sum of smaller EMGs (neurons). The signals received by each neurointerface sensor are a combination of signals from active oscillators in the brain. Thus, brain states are characterized by a combination of oscillator activity.This representation of brain signals as a sum of signals from nearly periodic signal generators makes it possible to describe the brain signal as a set of harmonic signals described by nearly periodic functions.

[0021] Thus, brain states can be described by a system of differential equations with constant coefficients for each sensor of the following form: y(t) = p a n

[0022]

[0023] < where a n (t) - polynomials, N is the number of oscillators.

[0024] Consequently, when similar brain states occur in the same person at different points in time (e.g., emotions of joy, states of concentration, meditative states, states of emotional uplift), a common information component is identified between the brain signals, represented by a certain number of brain oscillators and their characteristics (location, power, approximating function). Thus, brain signals from a given brain state (e.g., joy) will remain similar for one person over periods ranging from several days to several years.

[0025] The patented invention proposes to detect, record and store various user states in a library of mental states and subsequently activate the desired brain pattern in the user using feedback.

[0026] The first step involves recording and storing brain activity during states of mind. For example, when viewing photos or videos, listening to music, or solving problems in a state of high concentration ("flow"). Other brain states (joy, admiration, etc.) are recorded in a similar manner. Thus, the stored signal recordings correspond to a specific state and are stored in a library of mental states, which acts as a database.

[0027] The analog signals from the brain read by the electrodes (sensors) of the neurointerface based on the electroencephalogram or magnetoencephalogram have a range of 5-100 microvolts (μV) and 1-100 x 10 Л(-15) Tesla (T), respectively. According to biophysical data, the frequency spectrum of the brain's physiological electromagnetic signal lies in the range of 1-100 Hz. A frequency of 250 Hz is sufficient for signal analysis. The ADC converts the analog signal with a sampling frequency of 250 Hz. One measurement is packed into a 32-bit vector, of which 26 bits are informative and 6 bits are service. The signal error is 3 bits. Analysis of real data shows that all 26 bits are used significantly. The vector of values ​​is packed every 100 milliseconds (ms) in the data packing block and sent to the classification block.

[0028] In the second stage, the brain pattern stored in the mental state library and loaded as weights into the brain signal classifier is interactively activated using a neural interface and feedback. Subjectively, this is perceived as an experience similar to the stored state. This may be necessary for user rehabilitation (for example, in cases of amnesia) if they have lost the brain patterns corresponding to certain states. By loading brain states from the mental state library, the corresponding neural connections can be restored.

[0029] When implementing this method, the distance between the target brain state and the current brain state is compared. The target brain state is a pattern of brain activity described by the weights of a classifier (e.g., a neural network or a topological classifier). The current brain state, as data from the neural interface, is input to a classifier configured to identify the target state. The classifier produces an accuracy percentage, which indicates the similarity between the current state and the target state. The closer the current brain state is to the target state, according to the accuracy the classifier provides when analyzing the current state, the smaller the value of the selected metric (e.g., distance) between the states. When using a topological classifier, the closer (the smaller the distance) the current state (with the corresponding curve constructed using the unfolding method) to the hyperplane corresponding to the target state, the smaller the distance between the states.The user receives feedback in the form of an indicator for implementing biofeedback. For example, the indicator could be based on a color change ranging from red to blue. The bluer the object on the display, the closer the brain state is to the desired one. The color is calibrated using, for example, a classifier accuracy scale: 0% classification accuracy corresponds to red, 100% accuracy to blue, with a color gradient in the range. The color can be displayed, for example, on the screen of a mobile computing device (smartphone). When using a topological classifier in the classification unit of the neural interface, the similarity between the read brain signals and the brain signals for a specific state in the state library is assessed by identifying a common topological invariant and calculating the distance between the scan corresponding to the current state and the hyperplane corresponding to the desired state.To do this, the digitized physiological electromagnetic signal of the brain is represented as an object of topological space and the topological properties of the resulting time series scans and the number of signal generators (oscillators) are evaluated, having previously configured the classifier.

[0030] To identify a common topological invariant and the number of signal sources, the brain signal (time series) from each sensor is transformed into an object in a topological space and analyzed in this space. The analysis is based on the transition from a time series to a multidimensional unfolding—a piecewise linear curve in a multidimensional space whose points correspond to segments of the original series—and an analysis of the projections of this curve onto subspaces spanned by the principal components (vectors) of the corresponding scattering matrix.

[0031] A time series f = (fi, ..., fi) is a sequence of values ​​f(t), with a fixed time interval At.

[0032] Each digitized physiological electromagnetic signal from the brain, received as a time series (digitized sequence) from a sensor or sensors, must be assigned to one of m equivalence classes. Each equivalence class corresponds to a brain state being compared. This requires transforming the brain signal using the method described below and assessing whether the transformed signal fits into a topological space previously constructed for the brain state being compared. This is accomplished by performing steps S1–S3.

[0033] S1 Receive a digitalized discrete brain signal in the form of real-valued time series f 1 = (f 1 i, ..., f 1 N), ..., f 5 = (f®i, ..., FN) of fixed length N, where s is the number of neural interface sensors.

[0034] Next, a multidimensional (n-dimensional) Buchstaber unfolding is obtained from each time series.

[0035] If there is no a priori information about the expected number of signal sources, the maximum possible scan window n is selected. The recommended initial value is n = Q((N + 1) / 2), where Q is the integer part. This results in an n x p Hankel matrix, where p = N - n + 1, with rank < n.

[0036] In this case, the n-dimensional Buchstaber scan Xf = Xf(N, n) of the time series f = (fi, fn) in the n-dimensional Euclidean space is an oriented piecewise linear curve obtained by successively connecting the vectors Xi, X2, ..., Xp, where Xi = (fi, fi+1, ..., f i+nl).

[0037] From s time series (obtained from s sensors), time series scans are obtained in the form of Xf curves (namely, Xf1 .. Xfs curves - taking into account the index with the sensor number) with nodes that are described by the Hankel matrix.

[0038] Thus, the piecewise linear curve Xf of the Buchstaber scan of each time series of the corresponding sensor can be written in the form of a matrix, where the columns of the matrix are n-dimensional vectors or nodes of the curve Xf: Xi, X2, .... Xp, and the resulting matrix of p columns and n rows is a Hankel matrix of the form:

[0039] "

[0040] "

[0041]

[0042] Next, scattering matrices W are formed (one matrix for each sensor) of the Xf curves (obtained from the sensors), and eigenvalues ​​and eigenvectors are obtained for Buchstaber scans from each channel (sensor) of the neural interface data.

[0043] The rank r of the scattering matrix W of the Xf curves is calculated as the maximum number of eigenvalues ​​such that the remaining eigenvalues ​​can be neglected, taking into account the noise criterion.

[0044] In this case, the rank r (or the E-rank of r, when the eigenvalues ​​fall in the E-neighborhood) corresponds to the number of active signal sources - oscillators as follows: q = r / k, q is the number of active signal sources (oscillators) that generate the signal arriving at a particular sensor, k is the dimension of the space in which the signal arriving at a particular sensor is generated. When modeling a signal in a two-dimensional space, for example, reflecting the propagation of a signal over the surface of the cerebral cortex, the formula for the honor of brain oscillators is: m = r / 2.52 The Xf curves are projected onto the subspaces spanned by the first r principal components (eigenvectors whose eigenvalues ​​are higher than the value that was estimated as noise). The result is the Xf curves in r-dimensional space. This eliminates noise.

[0045] The Xf curves are projected onto the subspaces spanned by the principal components (vectors) of the corresponding scattering matrix. The resulting set of projections is fed to the classifier for classifying the neural interface data for brain state assessment.

[0046] 54 Carry out classification or clustering.

[0047] S4 1 At this step, the following proposed classifier can be used. Equivalence classes of time series are formed and a subspace (hyperplanes) is constructed for m equivalence classes {Ki,..., Km}, corresponding to m brain states. This step is performed on brain signal data for which the brain state is known. Each class corresponds to its own hyperplane with an E-neighborhood for each sensor of the neural interface. E is selected such that the resulting hyperplanes do not lie in each other's E-neighborhoods. The following algorithm can be used to implement the above:

[0048] The input of the algorithm is s time series {f1,..., fs}, fi = (fi,..., TN), grouped into the corresponding equivalence classes {Ki ,... , Km}. Then:

[0049] (1) We obtain the unfolding of s time series.

[0050] (2) We obtain the ranks of the piecewise linear curves, and the parameter r—the subspace dimension—corresponds to the rank (described in S1). We obtain r corresponding to the criterion—the maximum possible, taking into account the noise criterion, or the value necessary and sufficient for a given accuracy of brain state classification. Moreover, r can be different for each sensor.

[0051] (3) In the resulting subspace of dimension r, we construct m r-dimensional hyperplanes {Li, ..., Lm} with £-neighborhoods for each equivalence class {Ki, ..., Km}. We assign a value Ei to each hyperplane in such a way that the scans of one class lie in the £-neighborhood of the corresponding hyperplane Li, expressed, for example, by the corresponding confidence interval (10, 20, 3a). We obtain: {EI, ..., Em} are sets for m equivalence classes (and m x s sets for s sensors).

[0052] (4) Check whether {Li, ..., Lm} lie in E-neighborhoods of each other. for Li, L £ {Li Lm} do

[0053] for Lj, Lj 6 {Li, Lm}, ji do

[0054] if Lj e Ei is a neighborhood of Li then

[0055]

[0056] Other approaches are possible. For example, (a) pointwise removal from Li or (b) removal from Li of the corresponding unfolding and recalculation of Li and Ei.

[0057] end if

[0058] end for

[0059] end for

[0060] We get: updated values ​​{EI, ..., Em} .

[0061] OUTPUT: {Li,..., Lm}, {E1, ..., Em}.

[0062] The hyperplane subspaces {Li,..., Lm} with c-neighborhoods obtained for brain state equivalence classes enable the development of a classifier for assessing the membership of a brain signal in an equivalence class for the corresponding brain state. The accuracy of this classifier can be optimized using supervised learning.

[0063] The belonging of the studied brain signal from the neurointerface sensors is assessed based on the belonging of the Buchstaber scan of the corresponding time series to the c-neighborhood of r-dimensional hyperplanes corresponding to a certain class of brain states.

[0064] The similarity assessment of the signal under study and the selected signal in a certain state from the database (or recorded in real time from another individual) is carried out by classification.

[0065] The brain signal under study, represented as an n-dimensional Buchstaber scan Xf = Xf(N, n) of a time series f, belongs to the ξ-neighborhood of the r-dimensional hyperplane L if

[0066]

[0067] < E. IF the signal being studied belongs to the E-neighborhood of the corresponding class, then the brain signal being studied is similar to brain signals from this class. If, at the initial moment in time, the signal being studied is similar to the desired one, then the user is already in the desired state and there is no need to activate the brain pattern. If the feedback indicator is based on a color change that ranges from red to blue, and the bluer the color of the object on the display, the closer the brain state is to the desired one, then this case will correspond to the color blue. Red will correspond to the maximum possible distance between the Buchstaber scan of the time series and the hyperplane of the corresponding brain state (evalence class). Blue will correspond to the distance between the Buchstaber scan of the time series and the hyperplane less than E.The user must strive to transition from the initial state to the desired one, and the indicator will show their progress. If the user succeeds in this transition, the distance between the Buchstaber time series plot and the hyperplane will become smaller and continue to decrease until it becomes less than E. This will be accompanied by a color gradient from red to blue.

[0068] A neural network-based classifier can also be used. Training such a neural network-based classifier on EEG or MEG data from a neural interface can proceed as follows. First, the data is preprocessed so that an n-dimensional vector is formed from the multichannel EEG or MEG signal. This vector consists of a set of frequencies; parameters characterizing the placement of electrodes on the scalp; temporal parameters; statistical data; and other patterns. Second, a brain state label corresponding to this vector is added. Third, the resulting vectors are fed to the neural network input, and the neural network is trained on the generated labeled sample using backpropagation. The result of the trained neural network is a classifier that, with a specified accuracy, classifies the signal from the neural interface according to its correspondence to one of the mental states of the brain.In this case, the indicator color is calibrated using, for example, a classifier accuracy scale: 0% classification accuracy corresponds to red, 100% accuracy to blue, with a color gradient within the interval. Another type of indicator could be used, such as a scale from the current state to the desired one and a slider that moves along the scale depending on progress toward achieving the desired state. A variable pitch indicator could also be used.

[0069] Thus, through the use of a neural interface, classifier, and feedback, it is possible to activate a stored brain pattern corresponding to a specific brain state. Furthermore, a topological classifier can be implemented on programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and microcontrollers in the form of efficient matrix calculations, with an accuracy no worse than the required (no worse than other technologies). This avoids the energy-intensive training required by other technologies (e.g., neural networks) and the use of additional devices for such training and operation (in most cases, a graphics processing unit (GPU)), which also significantly increases energy consumption.Accordingly, FPGAs, specialized integrated circuits, and microcontrollers can be located within the neural interface itself, significantly increasing the speed of signal processing and determining the distance between the Buchstaber scan of a time series and the hyperplane of the corresponding mental brain state, as classification is performed within the neural interface itself, without remotely accessing the server. This enables real-time recognition of brain states while maintaining high accuracy.

[0070] The proposed invention also helps learn how to quickly achieve the desired state, for example, during meditation. Furthermore, this technology could in the future allow for the restoration (neurorehabilitation) of brain connections lost during thawing (following cryogenic freezing), provided such thawing does not result in significant neuronal loss.

[0071] Although the invention has been described with reference to the disclosed embodiments, it will be apparent to those skilled in the art that the specific experiments described in detail are provided merely for the purpose of illustrating the present invention and should not be construed as limiting the scope of the invention in any way. It should be understood that various modifications are possible without departing from the spirit of the present invention.

Claims

Formula 1. A method for activating a stored brain pattern, including the steps of: record brain activity when experiencing a certain state, using neural interface sensors placed on the user's head obtain a pattern of brain activity for a particular brain state; - store records of brain activity patterns for different brain states in a library of mental states; when the user selects from the library of mental states the required pattern of brain activity corresponding to the required brain state, the selected pattern is loaded in the form of weights of the neural interface classifier for the given user; receive signals from the user's brain through the neurointerface sensors placed on the user's head to determine the current state of the user's brain and the corresponding pattern of brain activity; - compare the current state of the brain, represented by the corresponding pattern of brain activity, with the required state of the brain, represented by the corresponding pattern of brain activity, and in the event of a discrepancy between the states, activate the stored brain pattern using feedback, wherein the feedback is implemented by means of an indicator that displays the distance of the current state from the required one.

2. The method according to claim 1, in which the recording of activity and the receipt of brain signals are carried out by means of a neurointerface, the sensors of which are electrodes or magnetic field strength sensors and are designed with the ability to record an electroencephalogram or magnetoencephalogram of the human brain.

3. The method according to paragraphs 1-2, in which a topological classifier or a classifier based on a neural network is used when comparing the current brain state with the required one.

4. The method according to paragraph 3, in which, when using a topological classifier, the current brain signal is transformed into an object of a topological space, the membership of the transformed brain signals in a topological subspace, previously constructed for the required brain state, is assessed, and if the transformed brain signal belongs to such a topological subspace, then the current state corresponds to the required one, otherwise the distance between the object of the topological space and the topological subspace is measured to determine the distance of the current state from the required one.

Images