Device for manipulating real objects in synchronization with brain activity

Abstract

The invention relates to a device for manipulating real objects (22) and to a hybrid real / virtual system for synchronizing neurophysiological activity with the perception and use of real objects, the device comprising: - an opaque housing (2) having an opening (3) through which at least one hand can be introduced for manipulating objects inside the opaque housing, - a screen (4) having at least two states, an opaque state in which the inside of the opaque housing is invisible and a visible state in which the inside of the housing is visible, - at least one detector (10) for detecting a thinking time of a user, - a switch (9) for controlling switching of the screen to the visible state in response to an instruction from the user, and - a processing unit (8) for collecting signals from the at least one detector (10) and the switch (9).

Description

Description Title of the invention: Device for manipulating real objects synchronized with brain activity technical field

[0001] The present invention relates to a device for manipulating real objects and a real / virtual hybrid system for synchronizing neurophysiological activity with the perception and use of real objects. Prior art

[0002] In general, the human mind perceives, understands, and acts within a complex environment, which implies a continuous and dynamic interaction between the individual and their ecosystem. Thus, to make relevant choices appropriate to an individual's goals, the brain must integrate multiple signals from this environment and provide continuous, real-time feedback. These information processing and integration mechanisms allow individuals to adopt behaviors adapted to a given situation and context. Despite this dynamic reality, the traditional approach to understanding the neural bases underlying human decision-making and behavior (whether in healthy individuals or those with neuropsychiatric disorders) involves conducting experiments in relatively static, often simulated, laboratory environments.This environment has an artificial dimension, since the researcher strives to control certain variables that cannot be controlled outside of it. While laboratory experiments assume that the behavioral and neurobiological measurements recorded in this context accurately reflect the complexity of cognitive processing, it is actually difficult to transpose the reality of the laboratory to the reality of the outside world.

[0003] It is now established that brain activity can differ from that elicited in response to the presentation of real objects compared to images depicting the same objects. Yet, many assessments use images representing stimuli that are not actually present, evoking indirect perceptions of the objects or scenes depicted. Images predominate over real objects in research because they are easy to create, easy to present quickly with precise timing on computer screens, and easy to control (for example, for low-level attributes like brightness). Furthermore, computer tools can easily be combined with neuroimaging to search for biomarkers related to risk-taking or impulsivity.

[0004]

[0005] However, images do not allow for real-world actions. They cannot be manipulated, nor do they allow for meaningful interaction with the represented stimulus. In other words, you can't hammer a nail with a picture of a hammer. It has been shown that behavior during a computer task does not mirror real-world behavior.

[0006] We know of the use of a computerized version of a card game called IGT for "Iowa Gambling Task" in English, in order to record the brain activity associated with each of the decision-making processes, see in particular the document "Behavioral and Neural Arguments of Motivational Influence on Decision Making During Uncertainty", Giustiniani et al., 2020, Frontiers in Neuroscience, June 2020 | Volume 14, Article 583.

[0007] However, people do not behave the same way with the virtual version as with the real one. They perform worse than on the real (ecological) version due to the development of a flawed strategy, which some authors have called the "interactive strategy." Participants adopt a strategy based on the software's assumed reactions. These studies have found that with the computer version of a neuropsychological test assessing decision-making ability, participants do not learn to preferentially choose advantageous decisions. Thus, it appears that with computerized tasks, subjects base their decisions on flawed strategies. This is especially true for older people, who were not born with computers, and in whom a lack of trust and preconceived ideas about computers lead to reactions that are not representative of their cognitive state.

[0008] This observation is common across many departments, including psychiatry, neurology, and geriatrics. For example, in patients with mild dementia, there is also a lack of correlation between screening tools for detecting cognitive impairment and validated computer tasks such as the IGT or BART (Balloon Analogue Risk Task). This raises questions about what these tasks actually reveal, the information they provide about the patient's decision-making difficulties, and the reliability of studies combining computer tools with neuroimaging to search for neuromarkers for diagnosis or prognosis of disease progression.

[0009]

[0010] We are familiar with the document “The real deal: Willingness-to-pay and satiety expectations are greater for real foods versus their images”, Carissa A. Romero, Michael T. Compton, Yueran Yang, and Jacqueline C. Snow, Program in Cognitive and Brain Sciences, Department of Psychology, University of Nevada, Reno, USA, Cortex 107, 78-91 [PubMed: 29233524]. This document describes a system for visualizing real objects in comparison to images. The device described is a wheel with compartments in which real objects are placed.

[0011]

[0012] Current known systems do not prevent the implementation of decisions based on erroneous strategies.

[0013] The present invention aims at a new device enabling the manipulation of real objects.

[0014] Another aim of the invention is to measure brain activity synchronously with the presentation and manipulation of real objects. Description of the invention

[0015] At least one of the objectives is achieved with a device for manipulating real objects and detecting moments of manipulation, this device comprising: - an opaque casing having an opening through which at least one hand can be inserted for the manipulation of objects inside the opaque casing, - a screen having at least two states, an opaque state in which the inside of the opaque casing is invisible and a visible state in which the inside of the casing is visible, - at least one detector to detect a user's reflection time, - a switch to control the screen's display state in response to a user command, and - a processing unit to collect signals from said at least one detector and the switch.

[0016]

[0017] With the device according to the invention, real objects are used which are revealed inside the case in a controlled and precise manner with the help of the detector and the switch.

[0018] Thus, a user can manipulate real objects, such as cards or dice. It is therefore easy to implement decision-making tasks with real objects, which are reproducible and can be synchronized with the brain activity of an integrated or independent tool.

[0019] By manipulating these real objects, perception and decision-making are more in line with real life, thus removing the barriers encountered with computer tools implementing virtual objects.

[0020] The window allows, when switched to visible state, the contents of the box to be revealed through it at a very precise moment initiated by the switch.

[0021] The detector's function is to measure the user's thinking time. is capable of measuring two distinct moments, for example the placement and lifting of the user's hand, representing the beginning and end of reflection.

[0022]

[0023] According to an advantageous feature of the invention, the screen can be a liquid crystal polymer window whose opacity is electronically controlled.

[0024] Thus, the switch between opaque and visible modes is almost instantaneous in response to a command from the switch. The user can therefore be positioned facing the window and not be able to see inside the enclosure when the window is opaque, and be able to see inside the enclosure when the window is visible.

[0025] Ideally, the opening in the casing is designed so that, during operation, the user cannot see inside the casing. The window is positioned in the path between the user's eyes and the objects being handled inside the casing.

[0026] One could, for example, consider a non-limiting embodiment in which the screen is a display screen equipped with a dedicated camera to show an image of the inside of the enclosure. This could be a tablet-type screen with a dedicated camera mounted on one side of the tablet inside the enclosure and a display screen on the opposite side, visible to the user. The image from the dedicated camera is displayed on the screen in response to a switch command.

[0027]

[0028] According to an advantageous feature of the invention, the switch can be a button for controlling a change of screen state following a user command. Any other control system can be considered, such as touch, sound, optical, capacitive, etc.

[0029] According to a preferred embodiment of the invention, for example a timer is provided to introduce, if necessary, a delay between the user's instruction and the screen switching to a visible state.

[0030] This delay can be a value between 0.8 and 1 second. And it can be variable, that is to say, determined randomly with each instruction from the user.

[0031] The advantage of such a delay is to avoid artifacts caused by movement and prevent the user from becoming accustomed to discovering objects inside the enclosure after a predefined and constant delay between the switch activation and the window becoming visible. This aims to capture the precise moment of discovery while maintaining the user's full attention. This delay also allows the user time to position themselves appropriately in front of the window.

[0032] According to an advantageous embodiment of the invention, said at least one The detector can be a capacitive surface used to detect the presence of a user's body part. In this case, this body part could advantageously be a hand, but it could also be a foot, an elbow, or something else.

[0033] This detector is designed to detect, for example, the moment a hand is placed on the ground and lifted. This detector can also be buttons or any detection device capable of detecting the beginning and end of an event, such as a phase of user reflection.

[0034] The device according to the invention may further include an infrared camera and infrared lighting for capturing an image of the object inside the case. This may be the dedicated camera or any other camera located inside the case. The infrared camera's function is thus to identify objects in the dark and, for example, to initiate object detection before the user activates a display on the screen. For example, when the object is a card or a die, the infrared camera performs the identification as soon as the card is turned over or the die is brought to rest. For this to work, the inside of the case is continuously illuminated with an infrared LED strip.

[0035] According to an advantageous embodiment of the invention, the processing unit can be configured to analyze the images acquired by the infrared camera.

[0036] The values ​​of the objects handled inside the case are automatically detected using a computer tool associated with artificial intelligence algorithms within the processing unit, with a temporal accuracy on the order of milliseconds.

[0037]

[0038] The device according to the invention may further include visible lighting for viewing the object in the housing through the screen.

[0039] The function of this lighting is to illuminate the objects being handled inside the case when the user requires viewing these objects.

[0040] According to an advantageous embodiment of the invention, the case is a bottomless case designed to be placed on a table. It is therefore a portable device that can be placed on a table.

[0041]

[0042] According to another aspect of the invention, a brain activity measurement system is proposed, comprising: - the device for manipulating real objects and detecting moments of manipulation as described above, - a sensor for measuring a user's brain activity, - a communication interface with the user, and - a computer connected to said device, to the sensor and to the interface, this computer being configured to sequence the course of a measurement of brain activity with manipulation of objects.

[0043] The user communication interface can be the computer screen and / or speakers and / or a microphone or any other input / output device.

[0044] The system according to the invention is a hybrid system that allows the user's behavior to be synchronized with the concomitant measurement of brain activity.

[0045] With the invention, it is therefore possible to measure brain activity during the presentation or manipulation of real objects, and to measure brain responses related to the perception of a product or to decision-making.

[0046] From a clinical perspective, in addition to providing a more precise, standardized behavioral tool for assessing pathological decision-making mechanisms, such a hybrid tool strengthens the search for biomarkers of decision-making and pathological behaviors in neuropsychiatry. This tool can be used routinely, which reinforces its value in the development of personalized medicine (precision medicine). Thus, when combined with neuroimaging methods, a tool according to the invention has the potential to predict risk upstream, offer a better assessment of patient progress over time, and help predict their clinical course.

[0047]

[0048] A method for measuring brain activity is also planned, comprising the following steps: - placing a user's hand on a capacitive surface to signal the start of a reflection, - lifting the hand from the capacitive surface to signal the end of reflection, - Manipulating objects inside an opaque case by reaching through an opening in the opaque case; this case being equipped with a screen having at least two states, an opaque state in which the inside of the opaque case is invisible and a visible state in which the inside of the case is visible, - Pressing a switch to control the screen so that it switches to the visible state, - visualization of the object, - measurement of the user's brain activity in synchronization with different moments of the steps performed above.

[0049]

[0050] Visualization can be performed directly by the user, who reports to a brain activity measurement system, or by camera that is directly connected to the brain activity measurement system. Description of the figures and methods of implementation.

[0051] Other advantages and features of the invention will become apparent upon reading the detailed description of implementations and embodiments. limiting, and the following attached drawings:

[0052] Figure 1 is a schematic view of an entire system according to the invention,

[0053] Figure 2 is a schematic exploded view of an example of components used to implement the invention, and

[0054] Figure 3 is a schematic view of a user during the use of the device according to the invention.

[0055] The embodiments described below are not exhaustive; in particular, variants of the invention may be implemented comprising only a selection of features described hereafter, isolated from the other features described, if this selection of features is sufficient to confer a technical advantage or to differentiate the invention from the prior art. This selection includes at least one preferably functional feature without structural details, or with only a portion of the structural details if this portion alone is sufficient to confer a technical advantage or to differentiate the invention from the prior art.

[0056] In the figures, elements common to several figures retain the same reference.

[0057] In Figure 1, the system 1 according to the invention comprises a housing 2 with opaque walls that prevent visibility into the interior. This housing 2 includes an opening 3 on a front face 2a. This opening 3 is located on a lower portion. The housing may include a base 2b, but it may also be without one when the housing is intended to be placed permanently on a table. The dimensions of the opening 3 are sufficient to allow the passage of two hands inside the housing 2 and to enable the manipulation of objects while blindfolded. The opening 3 is rectangular in shape, but any other shape is possible.

[0058] On the same front face 2a, above the opening 3, there is a liquid crystal polymer window 4 whose opacity can be controlled by an electrical pulse. In the opaque state, the inside of the enclosure is not visible. In the visible state, the inside of the enclosure and the objects to be handled are visible. To achieve this, a strip of visible LEDs 5 is provided inside the enclosure to illuminate the objects when the user makes the window transparent.

[0059] The housing 2 is also equipped with an infrared camera 6 on its top surface, but inside the housing, and with permanent infrared illumination 7. This enables automatic recognition of objects handled inside the housing. Recognition is performed by means of a processing unit 8 connected to window 4, camera 6, permanent infrared lighting 7 and LEDs 5.

[0060] The processing unit is equipped with hardware and software for image processing, input / output signal processing and the implementation of a process for measuring brain activity.

[0061] The system according to the invention is designed to precisely control the moment of visual discovery of an object by the user. A push button 9 is provided to control the change of state of the window after it is pressed by the user. The push button 9 is connected to the processing unit 8, through which the command is transmitted, but it can also be connected to the window either directly or via a timer that applies a random delay between pressing the button and the change of state of the window 4. The processing unit is also capable of applying this delay.

[0062] A capacitive sensor 10 is also included, designed as a hand rest. This capacitive sensor 10 is also connected to the processing unit, which is configured to detect when the user's hand is placed on the rest and when it is lifted. The time between these two events is referred to as the user's response time.

[0063] For measuring brain activity, an EEG 11 headset is planned. This electroencephalography (EEG) headset records the brain's electrical activity. It has non-invasive electrodes designed to be placed along the user's scalp.

[0064]

[0065] Figure 2 illustrates an example of an embodiment of the system according to the invention, with an exploded view of the processing unit 8. The invention notably allows for the capture of two phases the user goes through: the moment of their decision-making (what will their choice be?) and the moment they wish to reveal the object being manipulated in the case. The button 9 and the capacitive sensor 10 play a role in detecting these two phases.

[0066] In Figure 2, the capacitive sensor is a small capacitive surface measuring approximately 150 x 200 mm x 1 mm thick. An MPR121 chip detects the presence of a hand in contact with the copper surfaces; this chip is located beneath the varnish of the capacitive surface. The capacitive sensor has a microcontroller connected via a USB port to a central computer. The hand-placed / hand-lifted information is thus transmitted to the computer, which records it in a Lab Streaming Layer (LSL). During use, the user is instructed to place their hand on the capacitive surface while it is reflecting, and then to pick up the object with the same hand once the reflection is complete.

[0067] When an object is present in a compartment but still invisible, the user must press button 9, which is provided for this purpose. A delay intentionally random (between 800 and 1000 milliseconds) is added before triggering the lighting of the visible LEDs 5 and the desopacification of the window 4.

[0068] The purpose of the delay is to avoid artifacts due to movement, and the randomness is intended to prevent habituation.

[0069] Each event related to the button and the capacitive sensor is logged in an LSL stream. For example, an Arduino Leonardo type microcontroller (ATMEGA 32u4 microcontroller) can be used to link the button, the visible LEDs and the liquid crystal window.

[0070] To relieve the workload of computer 12, which is intended to run a brain measurement application, the operation of automatically identifying the object being manipulated in the case using the camera 6 is performed by a Jetson Nano® single-board computer 14. This mini-computer, measuring 100 x 80 x 29 mm, has CUDA processing units (128 cores) that make it efficient for image processing tasks. A Python program, using OpenCV and the pyzbar library, detects the manipulated object and transmits the information via the WebSocket protocol through a router 15 to the main computer 12. A small HDMI screen 16 allows monitoring of the image processing operation.

[0071] The mainframe computer 12 is also connected to a screen 17 of the brain measurement application's monitoring system. This mainframe computer 12 has the following functions, among others:

[0072] - to collect data from the various generated LSL streams: decision-making, button pressing, object perception, EEG, and

[0073] - to manage the kinematics of the experience.

[0074] A software program is responsible for both managing LSL flows and interacting with the user.

[0075] Physical events (button, capacitive surface, object revelation) are received in the form of pressed keyboard keys, which ensures minimal latency.

[0076] The measurement of neurophysiological activity is performed within the central computer 12, with events of interest synchronized with the execution. This allows for a detailed analysis of brain responses related to object perception and selection.

[0077]

[0078] We will now describe an example of using the device according to the invention within the framework of an IGT protocol. This task aims to measure the ability to implement a strategy and make decisions under conditions of uncertainty. The user begins the task with a fictitious sum of €2000 and aims to collect as much money as possible by maximizing their winnings. To do this, they have four piles of 100 cards each: A, B, C, and D. The user performs a total of 200 draws and can freely choose the pile from which he wishes to draw a card, at will.

[0079] In Figure 3, user 19 is seated in front of the device 2 according to the invention, which is placed on a table 21. User 19 is wearing headphones 11 that record the brain's electrical activity. Four decks of cards 22 are placed between user 19 and the device 2. The user begins by placing their hand on the capacitive surface and starts thinking. They then remove their hand from the capacitive surface to indicate that they have finished thinking. They choose a card without turning it over immediately. They place it at the bottom of the device 2 through the opening 3. Then they turn it over. The opaque screen 4 does not allow them to see the result immediately. The user must then press the button 9 to make the screen 4 transparent. After a delay of 0.8 to 1 second, the screen depolarizes and reveals the result of the choice for 2 seconds, so the user knows whether they have won or lost money.Camera 6 can be used to identify the card being handled inside the case as soon as that card is turned over by the user.

[0080] The user then removes the card and inserts the next one once the screen is off. During the various pauses, the total amount of the user's winnings is announced by the processing unit.

[0081]

[0082] We will now describe an example of using the device according to the invention within the framework of a GDT protocol. GDT is a dice game designed to assess an individual's decision-making in a risky gambling context. The objective of this task is to maximize winnings, with the participant starting with a fictitious capital of €1000. The participant will have to perform a total of 100 dice rolls, before which they must bet on one or more faces to appear. They can bet on 1, 2, 3, or 4 different faces, with the potential win or loss varying proportionally to the risk taken. The tasks can be coded in Python and implemented using the PsychoPy® software.

[0083] The software verbally asks the participant how many faces they wish to bet on (announcement time: 1 second). The user then makes their selection using a keypad located on the side of the device by pressing the key associated with the number of faces (1, 2, 3, 4). The software announces the different winning faces to the user (announcement time between 2.5 and 4 seconds depending on the number of faces chosen). The user then hears the instruction to "roll the die" and must roll a real die into the device through the opening 3. The polarized screen 4 prevents the user from seeing the result of the roll. The user then activates the switch 9 to reveal the face that appeared, with a random display delay between 0.8 and 1 second. The face is recognized by the camera located in the device, and the software announces "win" or "lose" (announcement time: 2 seconds) depending on the face obtained, followed by the sum. total of his winnings (announcement time of 1.5 seconds).

[0084] The device according to the invention is applicable in any field requiring close proximity to reality while also being able to measure brain activity. Some examples: - From a clinical perspective, the search for more precise biomarkers of decision-making disorders is needed. Current studies using virtual stimuli are far removed from the actual decision-making processes of patients. This concerns many populations: the elderly, people with Parkinson's disease, and psychiatric populations. - Still on the clinical level, study of dysfunctional brain mechanisms during the perception of real stimuli related to the pathology of interest (food, syringes, etc...). - From an industrial perspective, the device can help to understand the neural mechanisms involved in choosing one product over another. These studies, which are usually conducted with virtual stimuli and therefore yield uncertain results, would gain in accuracy with the use of real objects. - Still on the industrial level, the same problem arises when perceiving a product (for example, studies on packaging). Here again, the brain activations observed with the device would be as close as possible to reality and the expectations of manufacturers.

[0085] Of course, the invention is not limited to the examples just described. Many modifications can be made to these examples without departing from the scope of the present invention as described.

Claims

Demands 1. Device for manipulating real objects (22) and detecting moments of manipulation, this device comprising: - an opaque casing (2) having an opening (3) through which at least one hand can be inserted for the manipulation of objects inside the opaque casing, - a screen (4) having at least two states, an opaque state in which the inside of the opaque housing (2) is invisible and a visible state in which the inside of the housing is visible, - at least one detector (10) to detect a user's (19) reflection time, - a switch (9) to control the display (4) to be turned on in response to a user instruction, and - a processing unit (8) for collecting signals from said at least one detector and the switch, characterized in that the screen (4) is a liquid crystal polymer window whose opacity is electronically controlled.

2. Device according to claim 1, characterized in that the switch (9) is a button for controlling a change of state of the screen following a user instruction.

3. Device according to claim 2, characterized in that it includes a timer to introduce a delay between the user's instruction and the screen switching to a visible state.

4. Device according to claim 3, characterized in that the delay is a value between 0.8 and 1 second.

5. Device according to claim 5 or 6, characterized in that the delay is determined randomly at each instruction from the user.

6. Device according to any one of the preceding claims, characterized in that said at least one detector (10) is a capacitive surface for detecting the presence of a part of a user's body.

7. Device according to any one of the preceding claims, characterized in that it further comprises an infrared camera (6) and an infrared illumination (7) for capturing an image of the object in the casing (2).

8. Device according to claim 7, characterized in that the processing unit (8) is configured to analyze the images acquired by the infrared camera (6).

9. Device according to any one of the preceding claims, characterized in that it further comprises a visible light (5) for viewing the object in the housing (2) through the screen (4).

10. Device according to any one of the preceding claims, characterized in that the housing (2) is a bottomless housing intended to be placed on a table.

11. Brain activity measurement system comprising: - the device for manipulating real objects (22) and detecting moments of manipulation according to any one of the preceding claims, - a sensor (11) for measuring a user's brain activity, - a user communication interface (17), and - a computer (12) connected to said device, to the sensor and to the interface, this computer being configured to sequence the course of a measurement of brain activity with manipulation of objects.

12. Method for measuring brain activity comprising the following steps: - placing a user's hand on a capacitive surface to signal the start of a reflection, - lifting the hand from the capacitive surface to signal the end of reflection, - Manipulation of objects inside an opaque case by passing hands through an opening in the opaque case; this case being equipped with a screen having at least two states, an opaque state in which the inside of the opaque case is invisible and a visible state in which the inside of the case is visible, this screen being a liquid crystal polymer window whose opacity is electronically controlled - Pressing a switch to control the screen so that it switches to the visible state, - visualization of the object, - measurement of the user's brain activity in synchronization with different moments of the steps performed above.

Images