Information processing system and abnormality detection method therefor
The information processing system detects anomalies in cell culture experiments by comparing operational states with predefined criteria, facilitating anomaly detection and recovery in cell culture systems.
Patent Information
- Application Number
- PCT/JP2025/016937
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-05-08
- Publication Date
- 2026-11-12
AI Technical Summary
In cell culture experiments, it is difficult to detect anomalies due to the high degree of freedom in operations and the combination of various recipes, which can be changed during execution, making it challenging to determine abnormal states.
An information processing system comprising a robot device, imaging device, and control device that acquires and compares the state of the operation with predefined specific states to detect anomalies, and executes rescue recipes when necessary.
Enables effective detection and response to anomalies in cell culture experiments, allowing the system to return to normal operations and continue data acquisition.
Smart Images

Figure JP2025016937_12112026_PF_FP_ABST
Abstract
Description
Information Processing System and Anomaly Detection Method Thereof
[0001] The present disclosure relates to technologies such as an information processing system and an anomaly detection method thereof.
[0002] In the life science industry, modalities based on regenerative medicine and cell production are still in their infancy. Before researchers can obtain desired normal cells, there are various disturbance factors, and it is necessary to repeat a huge number of experiments at a time to search for appropriate intermediate environmental parameters. Note that a set of environmental parameters for producing or culturing cells is also referred to as a recipe.
[0003] As a general method used in such experiments for producing or culturing cells, a Bayesian experimental design method is known. For example, Patent Document 1 describes that, in a cell culture experiment, one or more execution parameter items that may affect the evaluation performance results and their search ranges are inferred based on a given feature extraction rule from past one or more types of execution performance results and evaluation performance results, and that a plurality of variable parameter values included in the search range are generated according to Bayesian optimization or the like on the basis of the past one or more types of execution performance results and evaluation performance results.
[0004] Japanese Unexamined Patent Application Publication No. 2023-035238
[0005] In cell culture experiments, it is necessary to repeat a huge number of experiments as described above. An unexpected situation may occur in the process of repeating such a huge number of experiments. However, the operation of the robot apparatus in cell experiments is performed according to various recipes. The various recipes may be combined, and furthermore, the recipe may be changed in accordance with a user's instruction in the process in which the recipe is being executed. As described above, in the cell culture experiment, since the content of the experiment has a high degree of freedom, it is difficult to determine what kind of operation or state is regarded as an anomaly. Therefore, it has been difficult to detect anomalies.
[0006] An object of the present disclosure is to provide a technology capable of detecting an anomaly that has occurred in an information processing system.
[0007] A typical embodiment of this disclosure has the following configuration. The information processing system of one embodiment includes a robot device, a database device that stores a first recipe indicating a first recipe for operating the robot device to perform a culture experiment of multiple cell samples, and specific state data indicating a specific state of an object to be operated on in the culture experiment, an imaging device that images the object to be operated on in accordance with the first recipe, and a control device that controls the robot device to execute the first recipe. When the object to be operated on in accordance with the first recipe, the control device acquires the state of the object to be operated as imaged by the imaging device, and if the acquired state of the object to be operated matches the specific state, it detects that an abnormality has occurred in the culture experiment.
[0008] Furthermore, an information processing system according to one embodiment includes a robot device, a database device that stores a first recipe indicating a first recipe for operating the robot device to perform a culture experiment on multiple cell samples, and specific state data indicating a specific state of an object to be operated on for the culture experiment, an imaging device that images the object to be operated on while it is operating according to the first recipe, and a control device that controls the robot device to execute the first recipe. The abnormality detection method of the information processing system includes the steps of: acquiring the state of the object to be operated on when it is operating according to the first recipe, and detecting that an abnormality has occurred in the culture experiment when the acquired state of the object to be operated on matches the specific state.
[0009] According to a representative embodiment of this disclosure, a technology can be provided that can detect abnormalities occurring in an information processing system. Other issues, configurations, and effects are shown in the embodiments for carrying out the invention.
[0010] This figure schematically shows an example of the configuration of an information processing system according to an embodiment. This figure shows an example of the appearance of the main part of the information processing system according to an embodiment. This figure shows an example of the configuration of a computer. This figure shows an example of the configuration of data stored in a database device. This figure shows an example of the configuration of a functional block of a cell culture experiment system. This figure shows an example of an abnormality judgment target state defined for each target operation. This figure shows an example of a video classification model when the target operation is the transport operation of a centrifuge tube and the abnormality judgment target state is the dropping of the centrifuge tube. This figure shows an example of the detection processing of the abnormality detection unit. This figure shows an example of rescue recipe data and rescue recipe judgment data. This is a flowchart showing an example of processing in a cell culture experiment system.
[0011] The embodiments of this disclosure will be described in detail below with reference to the drawings. In the drawings, the same parts are generally denoted by the same reference numerals, and repeated descriptions are omitted. In the drawings, the representation of components may not show their actual location, size, shape, extent, etc., in order to facilitate understanding of the invention.
[0012] In explanations, when describing program-based processing, the focus may be on the program, functions, or processing units. However, the main hardware component is the processor, or a controller, device, computer system, or other system composed of such a processor. A computer system, using resources such as memory and communication interfaces as appropriate, executes processing according to the program read into memory. This realizes the specified functions and processing units. A processor is composed of semiconductor devices such as a CPU / MPU or GPU. Processing is not limited to software program processing; it can also be implemented using dedicated circuits. Applicable dedicated circuits include FPGAs, ASICs, CPLDs, etc.
[0013] The program may be pre-installed as data on the target computer system, or it may be distributed as data from the program source to the target computer system. The program source may be a program distribution server on a communication network, or a non-transient computer-readable storage medium, such as a memory card or disk. The program may consist of multiple modules. The computer system may consist of multiple devices. The computer system may consist of a client-server system, a cloud computing system, an IoT system, etc. Various types of data and information are composed of structures such as tables and lists, but are not limited to these. Representations such as identification information, identifiers, IDs, names, and numbers are interchangeable.
[0014] (Embodiment) <Configuration of the information processing system>
[0015] Figure 1 is a schematic diagram showing an example of the configuration of an information processing system according to an embodiment. In this embodiment, the information processing system will be applied to a cell culture experiment system 10 for explanation. As shown in Figure 1, the cell culture experiment system 10 according to Embodiment 1 includes equipment located in an aseptic area SC1 where Grade A is assumed, equipment located in an aseptic area SC2 where Grade B / C is assumed, equipment located in a general control area SC3, and equipment located in other areas SC4.
[0016] The aseptic area SC1 is equipped with a robotic device 300, a detection device 400, and cell culture equipment 600. The aseptic area SC2 is equipped with a control device 100 and a terminal device 201. The general control area SC3 is equipped with a terminal device 200. The other area SC4 is equipped with a database device 500.
[0017] The equipment installed in the aseptic area SC1 is connected to the equipment installed in the aseptic area SC2 and the general control area SC3 via the local bus LB, enabling communication between them. Furthermore, the equipment installed in the aseptic area SC2 is connected to the equipment installed in the other area SC4 via the cloud network bus CB, enabling communication between them. It should be noted that the connections between equipment via the above buses are merely examples and are not limited to the above connection methods.
[0018] The control device 100 includes an operation control unit 101 and a detection control unit 102. The operation control unit 101 transmits control signals to the robot device 300 so that the robot device 300 performs operations related to the cell culture experiment. The detection control unit 102 transmits control signals to the detection device 400 or acquires detection data from the detection device 400 so that the detection device 400 detects a predetermined physical quantity or data.
[0019] The terminal device 200 has an input unit and an output unit. The terminal device 200 receives input settings and operations from a user, such as a researcher, via the input unit and transmits control signals or data corresponding to those input settings and operations to the control device 100 or the detection device 400. The terminal device 200 also acquires output signals or data from the control device 100 or the detection device 400 and outputs them to the user via the output unit. The terminal device 200 can perform input settings related to the operation control of the robot device 300 and / or operate the robot device 300 based on the user's input. Since the terminal device 200 is installed in the general control area SC3, the user can give instructions from a safe location. Alternatively, the user may use the terminal device 201 in the aseptic operation area SC2 to perform input settings related to the operation control of the robot device 300 and / or operate the robot device 300. The terminal devices 200 and 201 may be, for example, personal computers, tablet terminals, or personal digital assistants.
[0020] The robotic device 300 has various mechanisms, motors or actuators connected to these mechanisms, and operates them based on received control signals to perform various processes related to cell culture experiments. The robotic device 300 also exchanges information with the cell culture equipment 600 and handles the transfer of cell samples to be cultured.
[0021] The main steps in a cell culture experiment include, for example, the following:
[0022] Firstly, there is a process called "culture medium exchange." Cells use the necessary nutrients from the culture medium and metabolize those nutrients. Therefore, when the nutrient content decreases and the amount of metabolites increases in the culture medium, it is discarded and replaced with a new medium. This process is called "culture medium exchange."
[0023] Secondly, there is a process called "cell subculturing." This involves taking a portion of the cultured cells and transferring them to a new culture vessel. This process is also called transplanting. As cells divide repeatedly during culture, their number increases, eventually filling the culture vessel or depleting the nutrients in the culture medium.
[0024] Thirdly, there is a process called "cell seeding." Based on the measured number of cells / cell concentration, an appropriate amount of "culture medium" is added to dilute the cell suspension, and a predetermined amount of the cell suspension is transferred to a new culture vessel. This process is called "cell seeding." It can also be described as the process of adjusting the cell concentration to the desired level and uniformly distributing the suspension in the culture vessel. A "cell suspension" is a substance in which cultured cells (or microorganisms) are suspended in a liquid medium.
[0025] Furthermore, more specific tasks performed in various stages of the cell culture experiment, namely operations carried out by the robotic device 300, include the following:
[0026] Firstly, there is the operation called "pipette." This refers to the process of drawing up and dispensing liquid (cell suspension) using a pipette. In cell culture, the process of repeatedly drawing up and dispensing the cell suspension with a pipette sometimes also serves to loosen the cells in the suspension.
[0027] Secondly, there is the operation called "diffusion." This refers to the process of dispersing the cell suspension evenly across the bottom of the flask.
[0028] Thirdly, there is the operation called "dispensing." This refers to the process of accurately transferring a specified amount of liquid, such as culture medium, drug solution, or cell suspension, into a plate or tip using a pipette.
[0029] The detection device 400 detects information related to the operation of the robot device 300. The detection device 400 has an imaging unit 401 and a sensor unit 402.
[0030] The imaging unit 401 is located within the sterile operation area SC1 (laboratory). The imaging unit 401 captures images of operations performed by the robotic device 300, such as the actions corresponding to each step of a cell culture experiment performed by the robotic device 300. The imaging unit 401 is, for example, a digital video camera and outputs video data obtained through the capture.
[0031] The sensor unit 402 has one or more force sensors. The one or more force sensors are installed in various mechanisms of the robot device 300 and detect forces, pressures, strains, torsional forces, etc., applied to the various mechanisms as the robot device 300 operates, and output data corresponding to the detected physical quantities. The sensor unit 402 also has one or more environmental sensors. The one or more environmental sensors are installed in the aseptic operation area SC1 (laboratory). The one or more environmental sensors detect the ambient temperature, ambient humidity, atmospheric pressure, atmospheric composition, etc., in the laboratory and output data corresponding to the detected physical quantities.
[0032] The detection control unit 102 controls the time or period of imaging by the imaging unit 401, and, if necessary, coordinates with the operation control unit 101 to synchronize the timing of operations or processing.
[0033] The cell culture equipment 600 includes a culture unit 601, a measurement unit 602, and a drug solution placement unit 603.
[0034] The culture unit 601 includes an incubator and other components, and cultures cells in a constant temperature and humidity environment. The measurement unit 602 measures physical quantities representing the culture state of all or part of the cultured cells. The chemical solution installation unit 603 stores culture media as nutrients for culturing cells, chemical solutions used for measurement, consumables for adding chemical solutions, and instruments and consumables for aspirating and dispensing liquids such as culture medium or chemical solutions.
[0035] The database device 500 has a data storage area that stores various data and information sent from the terminal device 200, detection device 400, recipe determination device 550, cell culture equipment 600, etc., via the control device 100. The data stored in the database device 500 includes, for example, video data and environmental data obtained by the detection device 400. Data acquired by each of the above devices during the experiment and stored in the database device 500 is called intermediate data.
[0036] Furthermore, the database device 500 stores normal recipe data (first recipe data) indicating a normal recipe (first recipe) for operating the robot device 300 to perform culture experiments on multiple cell samples, video classification model data (specific state data) indicating a specific state of the operating target used in the culture experiment, rescue recipe data (second recipe data) indicating a rescue recipe (second recipe) for rescuing the operating target to return to the culture experiment using the normal recipe when an abnormality is detected, and rescue determination data indicating that the rescue recipe cannot be executed. Here, the specific state is a state that does not occur in the operating target during the execution of the normal recipe, and includes actions of the operating target that do not occur during the execution of the normal recipe.
[0037] The recipe determination device 550 sets recipes (normal recipe, rescue recipe) to be applied to each of multiple samples of cells to be cultured. In addition, to streamline the experiment, the recipe determination device 550 stores the priority order for verification of multiple types of environmental parameters that make up the recipe. The top N environmental parameters with the highest priority are called the determination recipe. The recipe determination device 550 refers to intermediate data from the database device 500 in order to modify and update the determination recipe according to a predetermined algorithm.
[0038] Cell culture experiments are performed through operations by the robotic device 300. The robotic device 300 works in conjunction with other devices or units to carry out the experiments. For example, the robotic device 300 may temporarily transfer the cultured cell workpiece to the culture unit 601 for retrieval, or transfer it to the measurement unit 602 for measurement before retrieving it and starting operations, or it may take in a drug solution from the drug solution installation unit 603 and use it on the cultured cells.
[0039] The priority of experiments performed by the robotic device 300, predetermined processes, etc., are instructed by the operation control unit 101. The operations to be performed by the robotic device 300 are predetermined by the recipe determination device 550 based on the recipe set for each sample, via the operation control unit 101. The recipe determination device 550 selects verification parameters, which are environmental parameters to be verified. Alternatively, it prioritizes sampling and instructs the operation control unit 101 to perform experiments according to that priority.
[0040] The recipe determination device 550 modifies the determined recipe based on a trend analysis of the state of cultured cells using the intermediate data stored in the database device 500. The recipe determination device 550 modifies past determined recipes using result indicators stored in the database device 500. If the artificial intelligence in the recipe determination device 550 finds a causal relationship between environmental parameters and expected culture results that is different from conventional methods in the intermediate data, it improves the weighting of the trend analysis in cell culture of the sample based on the intermediate data, that is, the criteria for determining the trend in the trend analysis.
[0041] Here, the main functions of the devices constituting the cell culture experiment system 10 according to the embodiment are summarized as follows. The detailed content of the functions will be described later.
[0042] When an operation target is operating according to a normal recipe (first recipe), the control device 100 acquires the state of the operation target imaged by the imaging unit 401, and detects that an abnormality has occurred in the culture experiment if the acquired state matches a specific state. Thereby, the control device 100 can detect an abnormality during execution of the normal recipe.
[0043] In addition, the control device 100 predefines a threshold value indicating the rate at which the acquired state matches the specific operation, and determines that the state matches the specific state when the acquired state of the operation target reaches a rate equal to or higher than the threshold value with respect to the specific state. Thereby, when the state of the operation target imaged by the imaging unit 401 approaches the specific state to a certain extent, the control device 100 can detect an abnormality. Further, for example, when the user changes the threshold value, the control device 100 can change the degree of determining the specific state for detecting an abnormality.
[0044] When receiving an execution of a rescue recipe (second recipe), the control device 100 executes the second recipe when it is determined that the rescue recipe is executable, based on the state indicated by the imaging data imaged by the imaging unit 401 and the state in which the rescue recipe indicated by the rescue determination data cannot be executed. Thereby, the cell culture experiment system 10 can return to the normal recipe. When determining that the rescue recipe is not executable, the control device 100 outputs a notification that the rescue recipe cannot be executed to the output unit of the terminal device 200. Thereby, it is possible to output a notification that the rescue recipe cannot be executed, in other words, that return to the normal recipe is not possible.
[0045] Figure 2 shows an example of the external appearance of the main parts of the cell culture experiment system 10 according to an embodiment. As shown in Figure 2, the cell culture experiment system 10 includes a control device 100, a terminal device 200, a robot device 300, an imaging unit 401, and a sensor unit 402. A table 620 is also provided, and on the table 620 are several instruments 610 that are to be processed (or may be operated when a recipe is executed) and operated by the robot device 300. The instruments 610 include centrifuge tubes 611, pipettes 612, containers 613, well plates 614, flasks 615, etc. The centrifuge tubes 611, pipettes 612, and containers 613 included in the instruments 610 are each operated by the robot device 300. The robot device 300, the several instruments 610, etc. are imaged by the imaging unit 401. In addition, the ambient temperature, ambient humidity, atmospheric pressure, and atmospheric composition in the laboratory are detected by an environmental sensor included in the sensor unit 402. Furthermore, pressure, torsional force, strain force, etc., generated in the various mechanisms constituting the robot device 300 are detected by force sensors included in the sensor unit 402.
[0046] The robot device 300 has a base that is placed on the floor. The robot device 300 also includes a rotatable joint connected to the base and an arm connected to the joint. Further joints and arms are connected to the end of the arm. A gripping part for gripping the instrument 610 is also connected to the tip of the arm. The robot device 300 moves and manipulates the instrument 610 using the joints, arm, gripping part, etc. The image captured by the imaging unit 401 is displayed on the output unit of the terminal device 200, for example, via the control device 100. The robot device 300, table 620, instrument 610, etc. are placed in the aseptic operation area SC1. By placing the instrument 610 etc. in the aseptic operation area SC1, it is possible to suppress the contamination of samples with foreign matter, etc. Other devices, such as devices for transporting the instrument 610 etc., may be placed in the aseptic operation area. In this example, the terminal device 201 is located in a sterile operation area SC2, which is different from the sterile operation area SC1, and can be operated by the user. The area where the terminal device 201 is located may be any other area with a lower cleanliness grade than the sterile operation area SC1, and is not limited to this example.
[0047] The control device 100, the database device 500, and the recipe determination device 550 can be implemented using a dedicated or general-purpose computer such as a PC (Personal Computer) or a workstation (WS, Work Station), or an electronic device equipped with a computer. In addition, the terminal device 201 can be implemented using a dedicated or general-purpose computer such as a PC, a smartphone, a mobile phone, a tablet terminal, or a PDA (Personal Digital Assistant), or an electronic device equipped with a computer. Details of the computer will be described later with reference to FIG. 3.
[0048] FIG. 3 is a diagram showing a configuration example of a computer. As shown in FIG. 3, the computer 90 includes a processor 91, a memory 92, a storage 93, an input unit 94, an output unit 95, a communication control unit 96, and a communication bus 97. The processor 91, the memory 92, the storage 93, the input unit 94, the output unit 95, and the communication control unit 96 are connected to each other via the communication bus 97 so as to be capable of transmitting and receiving signals to and from each other. A predetermined program PG is stored in the storage 93. The predetermined program PG includes a program for implementing an abnormality detection process to be described later (see FIG. 10). In the computer 90, the processor 91 reads out the program PG, develops it in the memory 92, and executes the program, so that the computer 90 functions as, for example, the control device 100, the database device 500, the recipe determination device 550, or an abnormality detection system that detects an abnormality during recipe execution of the cell culture experiment system 10.
[0049] FIG. 4 is a diagram showing a configuration example of data stored in the database device. As shown in FIG. 4, in the database device 500, for each cell sample to be subjected to a culture experiment, a sample number (No.) assigned to the sample, intermediate progress data related to the sample, an experiment result index, and the like are stored in association with each other. In addition, normal recipe data, moving image classification model data, rescue recipe data, and rescue recipe determination data are stored in the database device 500.
[0050] Figure 5 shows an example of the configuration of the functional blocks of the cell culture experiment system 10. As shown in Figure 5, the cell culture experiment system 10 includes a video feature extraction unit 701, an anomaly detection unit 702, an anomaly content identification unit 703, an anomaly response unit 704, an alert / dialogue display unit 705, and a rescue recipe execution unit 706. Furthermore, the cell culture experiment system 10 includes a time-series video feature model unit 711, an LSTM prediction unit 712, and a video classification model unit 721. In addition, the cell culture experiment system 10 includes a normal recipe execution instruction unit 731, a normal recipe execution unit 732, and an AI monitoring start / end processing unit 733. These functions are realized, for example, by the computer 90 described above.
[0051] The video feature extraction unit 701 extracts video features from real-time captured video captured by the imaging unit 401. The video feature extraction unit 701 outputs the extracted video features to the anomaly detection unit 702 and the LSTM prediction unit 712.
[0052] The anomaly detection unit 702 performs processing to detect anomalies based on the video features input from the video feature extraction unit 701 and the predicted video features input from the LSTM prediction unit 712. The anomaly detection unit 702 outputs the processing results and other information to the anomaly content identification unit 703. The processing information includes, for example, the error rate of the video features relative to the predicted video features.
[0053] The abnormality content identification unit 703 identifies the abnormal operation in which an abnormality was detected, based on the processing result input from the abnormality detection unit 702 and the video classification model indicating the abnormality target state stored in the video classification model unit 721. The abnormality content identification unit 703 outputs abnormality information indicating the content of the identified abnormality to the abnormality response unit 704.
[0054] The anomaly response unit 704 displays the content indicated by the anomaly information input from the anomaly content identification unit 703, that is, the detected anomaly, and, if necessary, a recovery recipe, on the alert dialog display unit 705. The process of identifying the recovery recipe to be displayed on the alert dialog display unit 705 will be described later. The anomaly response unit 704 also receives an execution instruction for the recovery recipe from the alert dialog display unit 705. Furthermore, the anomaly response unit 704 outputs an AI monitoring termination instruction to the AI monitoring start / termination processing unit 733.
[0055] The rescue recipe execution unit 706 executes the rescue recipe that is input from the alert dialog display unit 705 and received via the abnormality response unit 704. Once the rescue recipe is executed, the cell culture experiment system 10 returns to the normal recipe.
[0056] The time-series video feature model unit 711 receives data from the feature model updated by AI learning and updates the feature model. The updated feature model is output to the LSTM prediction unit 712. The feature model to be updated can be created by an AI learning unit (not shown) that performs AI learning in the same way as conventional technology.
[0057] The LSTM prediction unit 712 creates predicted video features based on the extracted video features input from the video feature extraction unit 701 and the feature model input from the time-series video feature model unit 711. LSTM (Long Short-Term Memory) is a type of neural network that is used by AI for machine learning. The LSTM prediction unit 712 uses the neural network to create predicted video features. The LSTM prediction unit 712 outputs the created predicted video features to the anomaly detection unit 702.
[0058] The video classification model unit 721 receives input data for the video classification model updated by AI learning and updates the video classification model. The updated video classification model is referenced by the anomaly content identification unit 703. The video classification model to be updated can be created by an AI learning unit (not shown) that performs AI learning in the same way as conventional technology.
[0059] The normal recipe execution instruction unit 731 receives a normal recipe execution instruction from the user and outputs the normal recipe execution instruction to the normal recipe execution unit 732. A normal recipe is an experimental recipe that is executed while being judged by the recipe judgment device 550. On the other hand, a rescue recipe is a recipe that returns to the normal recipe so that the experiment can continue if an abnormality is detected during the execution of a normal recipe.
[0060] When the normal recipe execution unit 732 receives an execution instruction from the normal recipe execution instruction unit 731, it outputs an AI monitoring start instruction and execution recipe information to the AI monitoring start / end processing unit 733.
[0061] When the AI monitoring start / end processing unit 733 receives an AI monitoring start instruction and execution recipe information, it starts AI monitoring of the operation of the robot device 300 based on the recipe. Also, when it receives an AI monitoring end instruction from the abnormality response unit 704, it ends the AI monitoring of the operation of the robot device 300 based on the recipe. Here, AI monitoring is the process of monitoring the operation of the robot device 300 using images captured by the imaging unit 401, etc.
[0062] <Operation target and abnormality detection target state>
[0063] Figure 6 shows an example of a table T100 of abnormality detection target states defined for each target operation. Here, the target operation is the target operation that is detected as a feature when a normal recipe is executed. The abnormality detection target state is the specific state described above when the target operation is being executed. Specifically, the abnormality detection target state is stored in the database device 500 as the video classification model described above. Note that the target operation and the abnormality detection target state associated with the target operation shown in Figure 6 are just examples, and states or operations that do not occur in the target operation during the execution of a normal recipe can also be defined as abnormality detection target states.
[0064] As shown in Figure 6, the target operations are defined as centrifuge tube transport, lid opening / closing, pipette operation, aspirator aspiration operation, and electric pipette tip attachment operation. The centrifuge tube transport operation is the operation in which the robotic device 300 transports the centrifuge tube 611 from one predetermined position to another. The lid opening / closing operation is the operation in which the robotic device 300 opens or closes the lid of the centrifuge tube 611. The pipette operation is the operation in which a small amount of liquid is measured, dispensed, or moved using a pipette. The aspirator aspiration operation is the operation in which aspiration and exhaust are performed using an aspirator. The electric pipette tip attachment operation is the operation in which aspiration and attachment of tips are performed by electronically controlling the electric pipette.
[0065] The transport operation of the centrifuge tube is associated with the following abnormal conditions: the dropping of the centrifuge tube 611 and abnormal gripping angle. The dropping of the centrifuge tube 611 is the movement of dropping from a state where it is being held by the robot device 300. An abnormal gripping angle is a state where the angle at which the centrifuge tube is being held by the robot device 300 is abnormal.
[0066] The lid opening and closing operation is associated with the lid open / closed state as a state to be detected as an abnormality. For example, the lid open / closed state is when the lid of the centrifuge tube 611 is not closed while it is being transported by the robot device 300, and when aspirating with a pipette or the like, the lid of the centrifuge tube 611 is not open.
[0067] Pipette operation is associated with two conditions that can be detected as abnormalities: abnormal tip attachment and abnormal dispensing position. Abnormal tip attachment occurs when the correct tip is not attached. Abnormal dispensing position occurs when the tip of the pipette is in a position other than where the liquid should be dispensed.
[0068] The aspirator suction operation is associated with the following abnormal conditions: dripping of aspirated liquid, abnormal suction position, and abnormal suction target. Dripping of aspirated liquid is when the aspirated liquid is dripping from the tip of the aspirator. Abnormal suction position is when the aspirator is drawing in liquid from a position that is not where it should be drawn in. Abnormal suction target is when the aspirator is drawing in liquid that is not where it should be drawn in.
[0069] The tip attachment operation of an electric pipette is associated with a condition that is subject to abnormality detection: tip attachment abnormality. Tip attachment abnormality occurs when the tip that should be attached is not attached.
[0070] <Video Classification Model>
[0071] Figure 7 shows an example of a video classification model when the target operation is the transport operation of centrifuge tubes and the abnormal condition to be judged is the dropping operation of centrifuge tubes. Centrifuge tubes 611a to 611e indicate the positions of the dropping centrifuge tubes 611, respectively.
[0072] As shown in Figure 7, the centrifuge tube 611a, which was being held by the robotic device 300, moves to the positions indicated by arrows AR1 (611b and 611c) and falls onto the table 620. After falling onto the table 620, it moves to the positions indicated by arrows AR2 (611d and 611e). In other words, it bounces off the table 620. The movement from position 611a to position 611e is an impossible movement during the transport of the centrifuge tube 611. This model of the movement of the centrifuge tube 611 is stored as a video classification model for the transport operation of the centrifuge tube 611.
[0073] Furthermore, during the descent of the centrifuge tube 611, the vector of the direction of movement differs between the position of the centrifuge tube 611c and the position of the centrifuge tube 611d. Such movement with differing vectors is not possible in the normal transport operation of the centrifuge tube 611. In this embodiment, the imaging unit 401 can acquire changes in the vector of the direction of movement of the moving object in order to image the centrifuge tube 611 in real time. This makes it possible to compare predicted video features and video classification models more accurately.
[0074] <Detection of abnormalities>
[0075] Figure 8 shows a table T200 of an example of the detection process of the abnormality content identification unit 703. As shown in Figure 8, the abnormality content identification unit 703 stores the elapsed time (s), prediction error rate (agreement rate with the abnormality target state), and abnormality detection result associated with the video "01_0001_mp4".
[0076] The video "01_0001_mp4" is an image captured by the imaging unit 401 during a transport operation in which the robot device 300 transports the centrifuge tube 611. In the example shown in Figure 8, the predicted error rate for this image is stored every 0.01 seconds, and the nature of the anomaly is identified based on the stored predicted error rate. Specifically, the predicted error rate of the image of the centrifuge tube 611 relative to the video classification model during the dropping operation of the centrifuge tube 611 shown in Figure 7 is calculated every 0.01 seconds. For example, from 0.01 seconds to 0.05 seconds, the calculated values are 15%, 14%, 16%, 21%, and 18% each time 0.01 seconds have elapsed. In this embodiment, the threshold for detecting an anomaly is set to 80% (threshold). Therefore, the anomaly detection result is normal up to 0.05 seconds. On the other hand, after 0.06 seconds, the calculated predicted error rate is 87%, which is above 80%. Therefore, an anomaly is detected after 0.06 seconds have elapsed.
[0077] <Rescue recipes and rescue criteria>
[0078] Figure 9 shows an example of table T300 containing rescue recipe data and rescue recipe judgment data. As previously described, a rescue recipe is a recipe used to return to the normal recipe when an abnormality is detected, that is, when an abnormality is detected. In Figure 9, the rescue recipe data is shown for the case of the centrifuge tube 611 dropping operation.
[0079] The dropping action of the centrifuge tube 611 is associated with rescue recipe data such as "recover the centrifuge tube 611 at the recovery location," "re-grasp the centrifuge tube 611," and "clean the leak location." The user operates the terminal device 200 to input the desired rescue recipe. By executing the rescue recipe, the system can return to the normal recipe from the state where an abnormality was detected. This allows the normal recipe of the cell culture experiment to be continued.
[0080] Furthermore, the rescue recipe data is associated with the mutual aid recipe judgment data. This is because there may be cases where the rescue recipe cannot be executed. The rescue recipe judgment data is used for this judgment. In the case of the centrifuge tube dropping operation, the rescue recipe judgment data is as follows: "Retrieve the centrifuge tube 611 to the retrieval location" is associated with "Detecting multiple broken parts of the centrifuge tube", "Re-grasp the centrifuge tube 611" is associated with "Cannot detect an image of the lid", and "Clean the leak location" is associated with "Cannot detect a leak".
[0081] For example, if multiple cracks are detected in the centrifuge tube 611, it is highly likely that the centrifuge tube 611 is broken. Therefore, the robotic device 300 cannot re-grasp the centrifuge tube 611. Consequently, it is difficult to return to the normal recipe. If an image of the lid of the centrifuge tube 611 cannot be detected, it is thought that there is a leak, and similarly, it is difficult to return to the normal recipe. Also, for example, if the centrifuge tube 611 is dropped, the lid may come off and a leak may occur on the table 620. In such a case, cleaning the leak location may be selected as a rescue recipe, but if there is no leak location, the rescue recipe cannot be executed. In this way, when a rescue recipe is selected, the state in which the rescue recipe cannot be executed is registered as rescue recipe determination data. In this case as well, for example, the prediction error rate for the rescue recipe determination data can be calculated, and if it is above a threshold, it is sufficient to detect that rescue is not possible.
[0082] <Anomaly Detection Processing> Figure 10 is a flowchart showing an example of anomaly detection processing in a cell culture experiment system. This processing is executed when a normal recipe is started in the cell culture experiment system 10.
[0083] (S901) First, the video feature point extraction process is performed. The image captured by the imaging unit 401 is input to the video feature extraction unit 701 in real time. For example, if the target operation is the transport operation of the centrifuge tube 611, the area around the centrifuge tube 611 is captured by the imaging unit 401, and the captured image of the area including the centrifuge tube 611 is input to the video feature extraction unit 701 in real time. The video feature extraction unit 701 calculates video features that use the centrifuge tube 611 as a feature. The video features are output to the anomaly detection unit 702 and the LSTM prediction unit 712.
[0084] (S902) Next, the AI judgment process is executed. The LSTM prediction unit 712 calculates predicted video image features from the video features and the model obtained from the time-series video features model unit 711. The anomaly detection unit 702 performs anomaly detection processing based on the video features input from the video features extraction unit 701 and the predicted video features input from the LSTM prediction unit 712, and outputs the processing result to the anomaly content identification unit 703. The anomaly content identification unit 703 creates, for example, the table shown in Figure 8 above. In other words, the anomaly content identification unit 703 stores the prediction error rate for the anomaly judgment target state (video classification model) of the video (video features) "01.0001.mp4" every 0.01 seconds.
[0085] (S903) Next, it is determined whether the prediction error rate is above a threshold. The abnormality content identification unit 703 determines whether the prediction error rate is above a threshold. If it is determined that the prediction error rate is not above a threshold (S903: NO), the process returns to step S901. In other words, the processing of the abnormality detection unit 702 and the abnormality content identification unit 703 continues.
[0086] (S904) If it is determined that the prediction error rate is above a threshold (S903: YES), an anomaly is detected. For example, in the example in Figure 8 above, after 0.06 seconds, the prediction error rate becomes 87%, which is above the threshold of 80%, so the anomaly content identification unit 703 detects an anomaly. In the example in Figure 8 above, the anomaly content identification unit 703 detects that an anomaly has occurred in the transport operation of the centrifuge tube 611 by the robot device 300, which coincides with a dropping operation. The anomaly content identification unit 703 outputs the details of the anomaly to the anomaly response unit 704.
[0087] (S905) Next, a recovery recipe is identified. The abnormality response unit 704 identifies a recovery recipe for the detected abnormality (specific condition) based on the table T300 shown in Figure 9 above. For example, if the detected abnormality is the dropping of the centrifuge tube, the abnormality response unit 704 refers to Figure 9 and obtains the following as candidate recovery recipes: "Move the centrifuge tube to the recovery location," "Grip the centrifuge tube again," and "Clean the leak location."
[0088] (S906) Next, it is determined whether or not the procedure is executable. The abnormality response unit 704 determines whether or not the acquired candidate rescue recipes can be executed based on the image captured by the imaging unit 401. For example, if multiple cracks are detected in the centrifuge tube 611, the rescue recipe "Grip the centrifuge tube again" is determined to be unexecutable. The abnormality response unit 704 determines that the procedure is executable if there is at least one executable rescue recipe, and unexecutable if there are no executable rescue recipes. If the abnormality response unit 704 determines that the procedure is executable, it outputs the detected abnormality and the rescue recipe to the alert dialog display unit 705. If the abnormality response unit 704 determines that the procedure is unexecutable, it outputs the details of the detected abnormality and the fact that there are no rescue recipes to the alert dialog display unit 705. If it is determined that the procedure is executable (S906: YES), the process proceeds to step S907. On the other hand, if it is determined that it is not executable (S906: NO), the process proceeds to step S909.
[0089] Furthermore, the anomaly response unit 704 instructs the AI monitoring start / end processing unit 733 to terminate AI monitoring. As a result, the processing of the AI monitoring start / end processing unit 733 ends. In the example shown in Figure 8, processing such as storing video and other data from 0.09 seconds onward, and calculating the prediction error rate are stopped.
[0090] (S907) A rescue recipe is selected. The alert dialog display unit 705 displays the detected anomaly and a rescue recipe to the user. The user selects a rescue recipe from the alert dialog display unit 705. For example, if the user selects the rescue recipe "Grip the centrifuge tube again" from the alert dialog display unit 705, an instruction to execute that rescue recipe is sent from the alert dialog display unit 705 to the anomaly response unit 704. Note that multiple rescue recipes may be selected in combination.
[0091] (S908) Next, the selected rescue recipe is executed. The abnormal response unit 704 outputs the rescue recipe execution instruction input from the alert dialog display unit 705 to the rescue recipe execution unit 706. The rescue recipe execution unit 706 then executes the rescue recipe. In the above case, based on the rescue recipe execution instruction "Grip the centrifuge tube again," the rescue recipe execution unit 706 operates the robot device 300 to grip the centrifuge tube 611. Then, the normal recipe execution instruction unit 731 outputs a normal recipe execution instruction to the normal recipe execution unit 732, and the normal recipe execution unit 732 outputs an AI monitoring start / end processing unit 733 with an AI monitoring start instruction and a normal recipe execution instruction. This allows the system to return to the normal recipe, and the cell culture experiment system 10 can continue to acquire experimental data for the cell culture experiment.
[0092] (S909) An alert is displayed. Because the rescue recipe cannot be executed, the alert dialog display unit 705 displays the detected anomaly and that the rescue recipe cannot be executed.
[0093] (S910) Operation is stopped. Since the rescue recipe cannot be executed, it is not possible to return to the normal recipe. Therefore, the cell culture experiment system 10 stops operating.
[0094] As explained above, the cell culture experiment system 10 can automate the detection of abnormalities in culture operations and the recovery process when abnormalities are detected. In other words, the cell culture experiment system 10 can detect abnormalities, execute a recovery recipe based on user instructions according to the detected abnormality, and return to the normal recipe even if an abnormality is detected. The cell culture experiment system 10 has more detailed effects, which will be explained below.
[0095] The cell culture experiment system 10 can detect abnormalities not by detecting a deviation from the normal state in which a standard recipe is executed, but by detecting the degree to which it matches a specific state (anomaly detection target state) that has been stored in advance. For example, the cell culture experiment system 10 can detect that an abnormality, namely a fall, has occurred when it detects that the centrifuge tube 611, which is the target of the operation, is bouncing on the table 620. In other words, assuming that the centrifuge tube 611 being transported by the robot device 300 has fallen, the cell culture experiment system 10 does not determine whether the centrifuge tube 611 is present or absent from the image taken by the imaging unit 401, or determine that it has fallen based on a change in the velocity of the centrifuge tube 611. Instead, it detects that the movement of the centrifuge tube 611 being imaged is approaching the movement of the bouncing centrifuge tube 611 stored as a specific state, and identifies it as a fall.
[0096] Furthermore, the cell culture experiment system 10 can perform a rescue operation if an abnormality is detected. The cell culture experiment system 10 determines whether it can return to the normal recipe, and based on the user's instructions, the cell culture experiment system 10 can return to the normal recipe. At this time, the user may operate the terminal device 200 to combine multiple selectable rescue recipes. For example, in the example of Figure 9 above, the user can combine "move the centrifuge tube to the retrieval location" and "clean the leak location" as rescue recipes. This allows the cell culture experiment system 10 to move the centrifuge tube to the retrieval location and clean the leak area, even if, for example, the lid of the centrifuge tube 611 opens due to a fall and a leak occurs. In this way, the cell culture experiment system 10 allows the user to freely combine rescue operations in the event of an abnormality detection at the time the abnormality is detected.
[0097] In addition, the cell culture experiment system 10 can also estimate the leakage area from the point where the centrifuge tube 611 falls, provided there is liquid at that point. For example, assuming a centrifuge tube 611 containing culture medium falls, the leakage area can be stored as a pattern based on the pre-controlled volume of contents in the centrifuge tube 611 and the viscosity of the culture medium. For example, the pattern of the leakage area may be associated with the rescue recipe data "Clean the leak location" in Figure 9. The abnormal response unit 704 may obtain the volume of contents and the viscosity of the culture medium from the fallen centrifuge tube 611, for example, from the database device 500, and the control device 100 may determine the leakage area. As a result, when "Clean the leak location" is selected as the rescue recipe, the cell culture experiment system 10 can clean the leakage area according to the type of liquid in the fallen centrifuge tube 611.
[0098] Furthermore, depending on the type of liquid contained in the centrifuge tube 611, the abnormal response unit 704 may automatically select to repeat the cleaning process several times. The wiping method and number of repetitions may be stored as patterns based on the viscosity of the liquid type, which has been managed in advance, and the abnormal response unit 704 may automatically select the necessary action.
[0099] Although embodiments of this disclosure have been described in detail above, the present invention is not limited to the embodiments described above and can be modified in various ways without departing from the gist of the invention. Each embodiment can be modified by adding, deleting, or replacing components, except for essential components. Unless otherwise specified, each component may be singular or plural. Combinations of each embodiment and its variations are also possible. Each of the above-described configurations, functions, and processing units may be implemented in part or in whole by hardware, such as by designing an integrated circuit, or by software, such as by a processor interpreting and executing a program. Data and information such as programs, tables, and files that realize each function can be stored in a recording device such as memory, a hard disk, or an SSD, or on a recording medium such as an IC card, an SD card, or a DVD.
[0100] 10...Cell culture experiment system (information processing system), 100...Control device, 200, 201...Terminal device, 300...Robot device, 400...Detection device, 401...Imaging unit, 402...Sensor unit, 500...Database device, 600...Cell culture equipment, 611 (611a to 611e)...Centrifuge tube
Claims
1. An information processing system comprising: a robotic device; a database device storing first recipe data indicating a first recipe for operating the robotic device to perform a culture experiment on multiple cell samples; specific state data indicating a specific state in which an object used in the culture experiment is operating; an imaging device for imaging the object operating according to the first recipe; and a control device for controlling the robotic device to execute the first recipe, wherein the control device acquires the state of the object being imaged by the imaging device when the object is operating according to the first recipe, and detects that an abnormality has occurred in the culture experiment if the acquired state of the object matches the specific state.
2. An information processing system according to claim 1, wherein the control device pre-determines a threshold value indicating the proportion of the acquired state that is determined to match the specific state, and determines that the acquired state of the operating target matches the specific state when the proportion of the acquired state to the specific state is equal to or greater than the threshold value.
3. An information processing system according to claim 1, wherein the database device further stores, when the abnormality is detected, a second recipe data indicating a second recipe for rescuing the system to return to the culture experiment according to the first recipe, and rescue determination data indicating that the second recipe cannot be executed; and the control device, upon receiving a request to execute the second recipe, executes the second recipe if it determines that the second recipe can be executed based on the state indicated by the imaging data captured by the imaging device and the unexecutable state indicated by the rescue determination data.
4. An information processing system according to claim 3, further comprising a terminal device having an output unit, wherein the control device outputs to the output unit that the second recipe cannot be executed when it determines that the second recipe cannot be executed.
5. An information processing system according to claim 1, further comprising a terminal device for giving instructions to the control device, wherein the culture experiment is performed in a sterile area, and the terminal device is located outside the sterile area.
6. An information processing system according to claim 1, wherein the object to be operated is a centrifuge tube held by the robotic device, and the state of the object to be operated is an operation indicating the falling of the centrifuge tube.
7. An information processing system according to claim 1, wherein the specific state is a state that does not occur in the target of operation during the execution of the first recipe.
8. An anomaly detection method for an information processing system, the information processing system comprising: a robot device; a database device storing first recipe data indicating a first recipe for operating the robot device to perform a culture experiment of multiple cell samples; and specific state data indicating a specific state in which an operating target used in the culture experiment is operating; an imaging device for imaging the operating target while it is operating according to the first recipe; and a control device for controlling the robot device to execute the first recipe, the anomaly detection method comprising: a step of acquiring the state of the operating target as imaged by the imaging device when the operating target is operating according to the first recipe; and a step of detecting that an anomaly has occurred in the culture experiment when the acquired state of the operating target matches the specific state.
