Guiding method based on endoscope operation data, guiding device, endoscope insertion practice method, insertion practice system, insertion practice method, and insertion practice program

Abstract

A guiding method based on endoscope operation data, comprising: a step for, with regards to a specific case, acquiring change information over time of operation information of an operator who operated an endoscope beforehand, and an endoscope image obtained by an imaging unit provided to an endoscope distal-end portion, and associating the operation information, the endoscope image, and the change information and recording the same as endoscope operation data, wherein in said step, the endoscope image of the specific case at a first timing and operation information regarding operation performed at the first timing are recorded; a step for referencing the endoscope operation data recorded in the recording step regarding the operation performed from the first timing; and an operation guide generation step for generating operation guide information on the basis of the referenced endoscope operation data.

Description

Guide method, guide device, endoscope insertion practice method, insertion practice system, insertion practice method, and insertion practice program based on endoscopic operation data

[0001] The present invention relates to a guide method, guide device, endoscope insertion practice method, insertion practice system, insertion practice method, and insertion practice program based on endoscopic operation data for assisting insertion practice into a lumen.

[0002] An endoscope is a device that inserts an imaging unit composed of an imaging device into the body or the like, irradiates illumination light from an attached illumination unit, and enables observation of a diseased part or the like that is difficult to view from the outside of the insertion site based on the imaging result (endoscopic image) of the imaging unit. The endoscope has an insertion part that is inserted into the body, and an imaging device is provided, for example, at the tip of the insertion part. In an examination using an endoscope, a doctor sequentially displays images acquired by the imaging device at the tip of the insertion part inserted into the body, adjusts the position of the tip of the insertion part while confirming the displayed images, and diagnoses the health condition and disease state of the body. Note that image information consisting of time-series image frames acquired during the process from the start of insertion of the endoscope into the body to the removal of the endoscope from the body may be digitized and recorded.

[0003] However, inserting an endoscope into the body is a task involving various operations, and it is difficult for non-experts to insert it into the target site. When a non-expert performs the insertion, appropriate insertion support to facilitate the insertion of the endoscope is desired.

[0004] As a technique related to such information collection, in U.S. Patent No. 7,894,648 (hereinafter referred to as Patent Document 1), a technique for evaluating the quality of an endoscopic procedure performed by inserting and removing an endoscope camera into a lumen is disclosed. In the proposal of Patent Document 1, digital video data including images of an endoscopic examination procedure is acquired, the digital video data is analyzed to acquire information regarding the movement of the camera passing through the lumen, and quality measurement criteria regarding the endoscopic examination procedure are calculated from the information on the movement of the camera. By using the proposal of Patent Document 1, it is conceivable to collect information necessary for assisting endoscope insertion and use it for insertion support.

[0005] Furthermore, Japanese Patent Publication No. 2009-219732 discloses a bending operation calculation device for practicing bending operations of an endoscope.

[0006] U.S. Patent No. 7,894,648; Japanese Patent Publication No. 2009-219732

[0007] However, even if the proposal in Patent Document 1 is used, it is not possible to collect sufficient information regarding endoscopic insertion operations necessary for appropriate insertion support. Furthermore, while Patent Document 2 uses a simulated operating unit to practice operations, it has the problem that it is not possible to conduct training that corresponds to various situations such as when actually inserting into a human body. The present invention aims to provide a guiding method, a guiding device, an endoscopic insertion practice method, an insertion practice system, an insertion practice method, and an insertion practice program based on endoscopic operation data that can generate guide information based on the operation data of an expert.

[0008] This invention makes effective use of image information that is easy for people to confirm, and uses this information to present appropriate operation guides using operation information between changes in the image. One aspect of the present invention provides a guiding method based on endoscopic operation data, which involves, for a specific case, acquiring time-dependent change information between operation information of an operator who has previously operated the endoscope and an endoscopic image obtained by an imaging unit provided at the tip of the endoscope, and recording the operation information, endoscopic image, and change information in association as endoscopic operation data, comprising the steps of: recording the endoscopic image of the specific case at a first timing and operation information related to operations performed from the first timing; referring to the endoscopic operation data recorded in the recording step with respect to operations performed from the first timing; and generating an operation guide based on the referenced endoscopic operation data.

[0009] A guide device according to one aspect of the present invention acquires time-dependent change information between operation information of an operator who has previously operated an endoscope for a specific case and an endoscope image obtained by an imaging unit provided at the tip of the endoscope, and records the operation information, endoscope image, and change information in association as endoscope operation data. The device includes an operation information recording unit that records the endoscope image of the specific case at a first timing and operation information related to operations performed from the first timing, and a guide generation unit that refers to the endoscope operation data recorded by the operation information recording unit with respect to operations performed from the first timing and generates operation guide information based on the referenced endoscope operation data.

[0010] A guiding method based on endoscopic operation data according to another aspect of the present invention includes the steps of: acquiring time-dependent change information between operation information of an operator who has previously operated the endoscope and an endoscopic image obtained by an imaging unit provided at the tip of the endoscope for a specific case; and recording the operation information, endoscopic image, and change information in association as endoscopic operation data, wherein the method includes the steps of: recording the endoscopic image of the specific case at first and second timings and operation information relating to operations performed from the first timing to the second timing; referencing the endoscopic operation data when the endoscopic image of the specific case at the first timing recorded in the recording unit changes to the endoscopic image at the second timing in the same case; and guiding the method to generate a plurality of guide information based on the referenced endoscopic operation data.

[0011] An endoscope insertion practice method according to one aspect of the present invention includes the steps of: acquiring time-dependent change information between operation information of an operator who has previously operated an endoscope for a specific case and an endoscope image obtained by an imaging unit provided at the tip of the endoscope; and recording the operation information, endoscope image, and change information in association as endoscope operation data, the steps of: recording the endoscope image of the specific case at a first timing and operation information related to operations performed from the first timing; referring to the endoscope operation data recorded in the recording step with respect to operations performed from the first timing; and generating an operation guide method that presents an endoscope operation guide method indicating the operation content for a predetermined unit of operation based on the referenced endoscope operation data. An insertion practice system according to one aspect of the present invention comprises: a first data acquisition device that acquires and records model operation data obtained by quantifying the amount of operation performed by an endoscope operator; a simulation device for simulating the insertion operation of the endoscope insertion section; a second data acquisition device that acquires simulation operation data obtained by quantifying the amount of operation performed by the simulation device; a guide generation unit that reads the model operation data and generates guide information indicating the operation content for predetermined operation units; and a guide presentation unit that provides information based on the guide information when the simulation device is operated.

[0012] An insertion practice method according to one aspect of the present invention is an insertion practice method for an insertion practice system having a first data acquisition device, a simulated operator, a second data acquisition device, a guide generation unit, and a guide presentation unit, wherein the first data acquisition device acquires and records model operation data obtained by quantifying the amount of operation performed by the endoscope operator, the simulated operator simulates the insertion operation of the endoscope insertion unit, the second data acquisition device acquires simulated operation data obtained by quantifying the amount of operation performed by the simulated operator, the guide generation unit reads the model operation data and generates guide information indicating the operation content for a predetermined operation unit, and the guide presentation unit provides a presentation based on the guide information when the simulated operator is operated.

[0013] An insertion practice program according to one aspect of the present invention involves a computer that acquires and records model operation data obtained by quantifying the amount of operation performed by an endoscope operator, and when the insertion operation of the endoscope insertion section is simulated, it acquires simulated operation data obtained by quantifying the amount of operation of the simulated operating device, reads the model operation data, generates guide information indicating the operation content for predetermined operation units, and performs a procedure to provide a presentation based on the guide information when the simulated operating device is operated.

[0014] The present invention has the effect of enabling the generation of guide information based on the operation data of skilled operators.

[0015] This is a block diagram illustrating an insertion practice system according to the first embodiment of the present invention. This is a block diagram illustrating an example of the specific configuration of the data processing device 20 in Figure 1. This is an explanatory diagram illustrating the state of the operating room. This is an explanatory diagram illustrating the operation judgment regarding pushing and pulling of the insertion part. This is an explanatory diagram illustrating the operation judgment regarding pushing and pulling of the insertion part. This is an explanatory diagram illustrating the operation judgment regarding pushing and pulling of the insertion part. This is an explanatory diagram illustrating the operation judgment regarding rotational operation (twisting) of the insertion part. This is an explanatory diagram illustrating the curvature amount judgment for bending operation of the insertion part. This is an explanatory diagram illustrating an example of the configuration of the simulated operation unit 51. This is a flowchart illustrating the generation of guide information. This is an explanatory diagram illustrating the generation of guide information. This is an explanatory diagram illustrating the generation of guide information. This is a flowchart illustrating insertion practice. This is an explanatory diagram illustrating insertion practice. This is an explanatory diagram illustrating insertion practice. This is an explanatory diagram illustrating insertion practice. This is an explanatory diagram illustrating a second embodiment. This is an explanatory diagram illustrating a second embodiment. This is an explanatory diagram illustrating a second embodiment.

[0016] Embodiments of the present invention will be described in detail below with reference to the drawings.

[0017] (First Embodiment) Figure 1 is a block diagram showing an insertion practice system according to the first embodiment of the present invention. In this embodiment, images from an endoscope and an external camera are captured in sync, and information related to endoscope operation (endoscopic operation data) is obtained for the generation of guide information by analyzing each image. In this case, in this embodiment, guide information is generated for each predetermined operation unit for the purpose of practicing insertion operations.

[0018] The lumens inside the body that are examined with an endoscope may be narrowed by sphincter muscles, bend three-dimensionally, be clogged with metabolites, or have raised lumen walls due to lesions. For this reason, it is not always possible to clearly visualize the lumen in the length direction (direction of hole connection) ahead of insertion from the endoscopic image acquired by the imaging unit, which is composed of an imaging device installed at the tip of the endoscope's insertion tube. There may be such difficult-to-insert sections with poor visibility in the middle of the lumen, or when entering a lumen from outside the body, entering another lumen from there, or when branching, as described above.

[0019] Therefore, the endoscope operator may need to proceed with insertion while searching for a path for the endoscope insertion site, even when the cavity ahead in the insertion direction is not necessarily visible (a dead end that appears to be a difficult insertion area). Moreover, the condition of the lumen varies from person to person, and even in the same person, the condition of the lumen differs depending on changes in the patient's position at the examination site, changes over time, and the circumstances at the time, making insertion of the endoscope insertion site not always easy.

[0020] Considering differences in examination equipment and usage methods, in most cases, sufficient experience is required from medical professionals to smoothly insert the endoscope into the lumen. Therefore, when a novice performs the examination, support from an experienced physician was necessary. However, with recent policies such as "training physicians with comprehensive clinical abilities," physicians are increasingly performing endoscopic examinations in addition to various other medical duties. Consequently, even trained physicians cannot always dedicate themselves to endoscopic operation while maintaining their operational skills.

[0021] To address these problems, one could consider using AI (artificial intelligence) to assist with the insertion of the insertion part, as a way to more efficiently transfer technology and provide an inspection method that even beginners can easily perform. However, collecting useful training data is difficult, and building inference models for such AI is not easy.

[0022] Therefore, in order to reliably acquire information on the characteristics of endoscopic operations performed by skilled physicians, this embodiment collects information indicating the operator's movements, such as the operator's posture, how they hold the endoscope in both hands, and hand movements, not only from endoscopic images but also from images taken with an external camera. Based on the collected endoscopic operation data, guide information for practicing insertion procedures is generated.

[0023] Generally, when trying to acquire a skill efficiently, partial practice is employed, where only a part of a series of actions is repeatedly performed. In insertion operations, such partial practice is considered to be extremely effective in acquiring the skill. Therefore, in this embodiment, guide information is generated for each operation unit, which is a part of the series of insertion operations when inserting the insertion part to the target site. It is believed that appropriate training becomes possible by using the guide information for each operation unit.

[0024] When inserting the insertion device into the target site, various operations are performed, not just pushing the device in, but also withdrawing it, twisting it clockwise or counterclockwise, and bending it up, down, left, or right to change its orientation. For example, an operation unit could be considered the period during which one of these operations is performed continuously. Alternatively, an operation unit could be considered the period during which multiple types of operations are performed simultaneously and continuously. For example, the period during which the surgeon twists and pushes the insertion device could be considered an operation unit. Furthermore, in the case of continuously pushing the insertion device, the surgeon may repeatedly move their arm in the pushing direction and then return it to its original position. Therefore, the operation unit may be determined based on the surgeon's movements. For example, the period during which the surgeon continues one type of movement could be considered an operation unit, or multiple types of movements performed simultaneously could be considered an operation unit.

[0025] In any case, the unit of operation should be set for each segment of operation that enables efficient practice such as repetitive practice. For example, the guide information generation unit 25, described later, can analyze the operations that caused the image changes between frames, and a single section with few types of operations that caused the image changes may be used as the unit of operation.

[0026] The insertion practice system 1 shown in Figure 1 includes a guide information generation system 10, an operation data recording unit 40, an endoscope operation practice unit 50, and a control unit 60. Each of the guide information generation system 10, operation data recording unit 40, endoscope operation practice unit 50, and control unit 60 may be composed of a processor using a CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), NPU (Neural Processing Unit), etc., and may operate according to a program stored in a memory (not shown) to control each part, or some or all of the functions may be realized by hardware electronic circuits. In Figure 1, the method of connecting each part of the insertion practice system 1 is not specified, but the guide information generation system 10, operation data recording unit 40, endoscope operation practice unit 50, and control unit 60 can be installed in locations separated from each other and connected by a network, making it applicable to insertion practice in remote locations. Since the operation unit is set as one division of operation that enables efficient practice such as repetitive practice, the amount of data to be transmitted is relatively small, and it is easy to apply to such remote practice systems.

[0027] (Guide Information Generation System) The guide information generation system 10 includes an endoscope operation monitoring system 11, a data processing device 20, a target part classification unit 30, and a basic information acquisition unit 31. The endoscope operation monitoring system 11 includes an endoscope system 12, an external camera 13, an UPD information acquisition unit 14, and a synchronization generation unit 15. Although Figure 1 shows one external camera 13, multiple external cameras 13 may be used.

[0028] The endoscope system 12 includes an endoscope (not shown). The endoscope has a long, slender insertion section that is inserted into the body. An imaging unit 12a is provided at the tip of the insertion section. The imaging unit 12a, as an imaging device, includes an image sensor such as a CCD or CMOS sensor, and obtains an imaging signal by photoelectric conversion of an optical image from the subject. A curved section (not shown) is also provided at the tip of the insertion section.

[0029] The endoscope system 12 is configured to manage image frames obtained from the endoscope imaging unit 12a in a time series on a time axis, enabling the acquisition of time-dependent change information of images obtained by the imaging unit 12a located at the tip of the endoscope for a specific case. The synchronization generation unit 15, described later, manages which image frame was obtained at what timing for which case (associated with the subject, operator, corresponding UPD information, and equipment information), and can synchronize with operation information based on the endoscope operation data described later. In Figure 1, the operation data recording unit 40 conceptually shows operation information between image frames, indicating what operations occurred between frames. This information is based on the endoscope operation data described later, but it may also utilize analog continuous information.

[0030] Considering that the procedure differs from case to case, it is better to distinguish cases by classifications such as who (what kind of subject (race, sex, body type)), when, and where (what kind of testing facility, laboratory, operating room, what kind of doctor (body type)). Furthermore, there are differences in how the endoscope is used (operation) depending on the type of examination or treatment of which organ and part for a specific disease, and we envision a classification of cases that can handle these differences separately. In Figure 15 described later, it is envisioned that the type of operation can be selected according to the case based on representative items such as subject, organ, site, and doctor, but this is an example.

[0031] An operating section is provided at the proximal end of the insertion section. The operating section is equipped with various operating devices, such as a dial (angle knob), for bending the curved section (angle operation), and the curved section can be bent by operating the dial. The operating section is also equipped with switches (not shown) for operating various medical devices (not shown), such as air supply / water supply / suction devices. A forceps channel for inserting treatment instruments may also be provided. Sensors or detection devices may be used to determine what operations were performed by the operator at what time in which case.

[0032] The imaging unit 12a receives the optical image of the subject and obtains an imaging signal from within the body through photoelectric conversion. The imaging signal obtained by the imaging unit is supplied to the internal video information generation unit 12b. The internal video information generation unit 12b generates an endoscopic image by performing predetermined signal processing on the imaging signal. The internal video information generation unit 12b outputs the generated endoscopic image (hereinafter also referred to as internal image, internal video, or first time-series image) as internal video information to the data processing device 20.

[0033] The external camera 13 has an imaging unit 13a. This imaging unit 13a includes, for example, an image sensor such as a CCD or CMOS sensor, and obtains an imaging signal by photoelectric conversion of an optical image from a subject. The imaging signal obtained by the imaging unit 13a of the external camera 13 is supplied to the external video information generation unit 13b. The external video information generation unit 13b generates an image by performing predetermined signal processing on the imaging signal acquired by the external camera 13. The external video information generation unit 13b outputs the generated image (hereinafter also referred to as the external image, external video, or second time-series image) as external video information to the data processing device 20.

[0034] This external camera 13 monitors the operator's actions in response to changes in the endoscopic image obtained by the endoscopic imaging unit 12a, and may be a single camera or multiple cameras. Alternatively, the determination of the operator's actions, which is handled by the external camera 13, may be replaced by sensors provided on the endoscopic control unit. In other words, the guide information generation system 10 acquires time-dependent change information between the operator's actions when operating the endoscope and the endoscopic image obtained by the imaging unit 12a located at the tip of the endoscope for a specific case, and records the operation information, endoscopic image, and change information in association as endoscopic operation data (hereinafter also simply referred to as operation data). The operation information is also managed using clock information (not shown) to indicate when and what kind of actions were performed.

[0035] The synchronization generation unit 15 is capable of synchronizing the imaging of the imaging unit 12a of the endoscope system 12 with the imaging of the imaging unit 13a of the external camera 13 by utilizing the above-mentioned clock information. The synchronization generation unit 15 can synchronize the imaging units 12a and 13a of the endoscope system 12 and the external camera 13 by employing various synchronization methods. For example, the synchronization generation unit 15 may synchronize the internal video and external video by generating a common trigger for the endoscope system 12 and the external camera 13 and controlling the imaging units 12a and 13a of the endoscope system 12 and the external camera 13 to start imaging according to this trigger. Alternatively, for example, the synchronization generation unit 15 may assign a timestamp to the internal video and the external video during imaging and synchronize the internal video and the external video by referring to the timestamp. Synchronization may also be performed manually using a clapperboard or the like. The term "synchronization" here refers to "matching and synchronizing the timing of operations," assuming a situation where the output image frames from the two imaging units 12a and 13a are obtained and aligned almost simultaneously. However, as the images captured and displayed in each frame of the imaging units 12a and 13a change moment by moment, it is important that the relationship between the acquisition timing of the output images from the two imaging units 12a and 13a is understood, as it is sufficient to be able to analyze from the images of the imaging units 12a and 13a what operations are causing the changes. Furthermore, the frame rate of the image frames from the imaging units 12a and 13a may be high, or even if the frame rate is low, it is acceptable as long as analysis is possible through frame interpolation, etc. In this way, the guide information generation system 10 acquires time-dependent change information between the operator's operation information for operating the endoscope and the endoscopic image obtained by the imaging unit 12a located at the tip of the endoscope for each specific case, and obtains endoscopic operation data that associates the operation information, endoscopic image, and change information.

[0036] The synchronization generation unit 15 may generate synchronization information necessary for synchronizing the internal video and the external video and output it to the data processing unit 20, or it may add the synchronization information to the internal video and the external video respectively before outputting them to the data processing unit 20.

[0037] The UPD information acquisition unit 14 acquires an image (UPD image) of the insertion shape of the insertion section obtained by a UPD device (endoscopic insertion shape observation device) (not shown). The UPD information acquisition unit 14 provides the acquired UPD image to the operation data recording unit 40 for recording. This UPD information is also associated with the operation information of the operator operating the endoscope and the time-dependent change information of the endoscopic image obtained by the imaging unit 12a provided at the tip of the endoscope by the guide information generation system 10 for each specific case.

[0038] The basic information acquisition unit 31 acquires basic information about the operator performing the examination, the subject's profile, and the examination details in the endoscopy system 12. The basic information acquisition unit 31 provides the acquired basic information to the operation data recording unit 40 for recording.

[0039] The target classification unit 30 generates target classification data based on user operations, indicating what type of target the operation data output from the guide information generation system 10 pertains to, and outputs it to the operation data recording unit 40. The target classification data can include medical records, CT scans, MRI images, etc. Considering differences in the type of endoscope used and surgical procedures, there may be cases where the operation data acquired by the data processing device 20 cannot be used. For example, comparing operation data from an upper gastrointestinal endoscopy with operation data from a lower gastrointestinal endoscopy may be meaningless. By using the target classification data generated by the target classification unit 30, it becomes possible, for example, to refer to operation data for the same surgical procedure.

[0040] With this configuration, the guide information generation system 10 acquires time-dependent change information between the operator's operation information (operation information of an operator who has previously operated the endoscope) and the endoscopic image obtained by the imaging unit 12a located at the tip of the endoscope, and records the operation information, endoscopic image, and change information in association. As shown in Figure 1, the data in the operation data recording unit 40 is organized and recorded in a database for each case, so it can be referenced with a simple selection operation. This database may be expanded to include more cases and become big data that can handle various situations. Using the data collected here, it may be possible to generate information on the relationship between images and operations for unknown cases (parts). The more cases that are collected, the better the predictions (RAG (Search Expansion Generation)) for cases become. This makes it possible to select and propose the most suitable information for dealing with unknown cases.

[0041] Furthermore, by considering patients from different countries and regions (which may be classified and recorded by gender, age, and body type) and doctors from different countries and regions, and obtaining data that allows for classification, it becomes possible to generate guides tailored to individual situations according to the classification. Clinics, national hospitals, and university hospitals differ not only in the doctors operating the system but also in the medical professionals assisting them and the equipment used. Due to these differences in circumstances, even for the same case, there may be multiple models corresponding to the case, depending on the differences in circumstances. In other words, the model best suited to the operator may be recommended based on the operator's profile input, such as workplace, skill level, experience, country, and region. Also, for some diseases, there may be a large number of subjects of a specific gender or age group, so it may be possible to start with the most frequent cases for practice and, if successful (as skill level improves), recommend rarer cases. In other words, it is possible to customize recommended practice methods and recommended guide methods. The database may also be constructed to allow selection of cases according to whether speed is a priority or whether time can be taken, the degree of pre-processing, and whether or not procedures are performed during the examination.

[0042] Since it takes time and effort to input all such diverse conditions, it may be possible to input the profile of the operator to be used, display a selection screen with the number of icons narrowed down from that information, and enable case selection. The parts of the organ can also be classified and organized into difficult parts and non-difficult parts, and recommendations can be made according to the profile of the operator. Alternatively, the operation history and achievement level (evaluated by the difficulty of the practice target, the number of OKs and NGs, the time until an OK is achieved, etc.) can be recorded, and recommendations can be made based on the results.

[0043] FIG. 2 is a block diagram showing an example of a specific configuration of the data processing device 20 in FIG. 1. The data processing device 20 as the first data acquisition device includes a first time-series image analysis unit 21, a second time-series image analysis unit 22, an information adoption unit 23, a time-series operation information generation unit 24, and a guide information generation unit 25. Each part of the data processing device 20 may be configured by a processor using a CPU, GPU, FPGA, NPU, etc. The data processing device 20 may operate according to a program stored in a memory (not shown) to control each part, or may implement some or all of the functions by an electronic circuit of hardware.

[0044] The first time-series image analysis unit 21 is given internal image information and analyzes the internal image as the first time-series image. The first time-series image analysis unit 21 converts, for example, an image change of the internal image into information corresponding to the operator's operation (first operation information), such as angular information. In this way, the first time-series image analysis unit 21 obtains the first operation information corresponding to the operator's operation based on the image change of the first time-series image by analyzing the first time-series image.

[0045] In addition, the first time-series image analysis unit 21 also determines the observation extraction site of which site the part being imaged by the imaging unit is. For example, the first time-series image analysis unit 21 can perform observation site determination by referring to a database (not shown) for patterns such as characteristic colors, shapes, and blood vessels visible on the surface of the part. The first time-series image analysis unit 21 may also perform site determination by adopting a known AI (artificial intelligence) for site determination.

[0046] The second time-series image analysis unit 22 is given external image information and analyzes the external image as a second time-series image. The second time-series image analysis unit 22 converts, for example, image changes in the external image into information corresponding to the operator's operations (second operation information), such as distance and angle. In this way, the second time-series image analysis unit 22 obtains second operation information corresponding to the operator's operations based on the image changes in the second time-series image through the analysis process of the second time-series image.

[0047] The information acquisition unit 23 selects information from the first and second time-series images necessary for collecting information related to operations on the endoscope. For example, the analysis results of the second time-series image may be necessary for specific operations within the endoscope. In this case, the information acquisition unit 23 selects the analysis results of the second time-series image to obtain information related to the operation of that operation. Also, in cases such as when the insertion part does not move in the appropriate direction or when the operator is located outside the shooting range of the external camera 13, there may be no image portion in the internal or external image necessary for collecting information related to endoscope operations. In this case, the information acquisition unit 23 selects and acquires an image frame that contains the image portion necessary for collecting information related to endoscope operations. The time-series operation information generation unit 24 generates time-series operation information indicating the content of the operator's operations based on the analysis results of the information acquired by the information acquisition unit 23 and records it as operation data in the operation data recording unit 40.

[0048] The guide information generation unit 25 as a guide generation unit reads the operation data recorded in the operation data recording unit 40 and generates guide information indicating the operation content for each operation unit. For example, the guide information generation unit 25 divides the operation data recorded in the operation data recording unit 40 for each operation unit, and identifies the first endoscopic image (hereinafter referred to as the first image) and the last endoscopic image (hereinafter referred to as the second image) at the head of each operation unit. For example, a "continuous operation by the same site" continuously performed from the first image to the second image becomes an operation unit. By using this operation unit as a guide for the practice unit of the repetitive practice of the technique, effective acquisition of the insertion technique becomes possible. That is, the guide information generation unit 25 converts the operation data into guide information by dividing the operation data for each operation unit. The guide information generation unit 25 records the operation data from the time series operation information generation unit 24 in the operation data recording unit 40 so that the operation unit, the first image, and the second image can be identified.

[0049] Further, the change from the first image to the second image occurs according to a temporal change and a timing change. Even if these two images are not defined, it is also possible to generate guide information only from the first image. That is, by using information that is easy to visually confirm, such as the first image, to present the state at that time and show the user as a guide what operations should be performed from there (operation information is not as easy to confirm as image information), it is excellent and easy to use in terms of the user interface.

[0050] Such operation guides can be applied to operation guides in various situations. When applied to the guide for insertion, the second image can also be defined as an image in which the path of insertion can be seen. That is, the second image does not have to be strictly defined or determined at the time of guidance, and may be an image corresponding to a preferable state obtained as an assumption according to the purpose of the guidance.

[0051] Figure 3 is an explanatory diagram for explaining the state of the operating room.

[0052] As shown in Figure 3, a patient 71 is lying on a bed 70 in the operating room. Figure 3 shows the surgeon 73 standing beside the bed 70 and attempting to insert the insertion device 79 into the patient's 71 anus with his hand 74. Three cameras 77a to 77c (hereinafter referred to as camera 77 when there is no need to distinguish between them) corresponding to the external camera 13 in Figure 1 are positioned at three locations near the bed 70. Each camera 77 can capture images in sync with each other by the synchronization generation unit 15 in Figure 1. Each camera 77 captures images of the patient 71 and surgeon 73 from different positions, allowing for images of the surgeon's 73 overall posture, arms, right wrist and fingers, right elbow, left fingers, and the patient's 71 posture from various angles. Although Figure 3 shows an example using three cameras 77 as the external camera 13, the number of cameras is not limited to this; it may be two or fewer, or four or more.

[0053] Furthermore, display devices 75 and 76 are positioned so that the patient 71 can see them. Although Figure 2 shows an example where two display devices 75 and 76 are used, only one of the display devices may be used. Also, the layout of the operating room is just one example, and the positions and number of cameras 77 and display devices 75 and 76 can be set as appropriate.

[0054] As described above, the endoscope 10 has an elongated insertion section (reference numeral 79 in Figure 3) that is inserted into a body cavity, and an operating section (not shown) provided at the rear end of the insertion section 79. The insertion section 79 also has a flexible section adjacent to the tip of the insertion section 79, and the flexible section can be bent using a dial provided on the operating section. The subject is imaged by the imaging unit 12a provided at the tip of the insertion section 79, and the imaged image (internal video) is obtained by the internal video information generation unit 12b. This internal video is supplied to the display devices 75 and 76 in Figure 3 and displayed on the display screen.

[0055] The operator, such as a physician, operates the endoscope 10 and inserts the insertion section 79 of the endoscope 10 into the patient 71. While viewing the endoscopic image (internal image) on the display devices 75 and 76, the operator operates the dials and other controls on the control unit to bend the curved section or move the insertion section 79 forward and backward, inserting the tip of the insertion section 79 to the area to be observed.

[0056] Specifically, during the process of inserting the surgeon's insertion device 79 into the body, internal images are obtained by imaging the internal conduits and other structures into which the insertion device 79 is inserted. In addition, during the process of inserting the surgeon's insertion device 79 into the body, external images are obtained by imaging various parts of the surgeon while the insertion device 79 is being inserted.

[0057] The surgeon's insertion operations are generally performed on the insertion section 79 itself, that is, the "tube," and the dial on the control section. Operations on the tube, which is the object of manipulation, include pushing and pulling the tube into the body and twisting the tube. Possible means of applying such pushing, pulling, and twisting to the tube include insertion using the surgeon's entire body, that is, through the surgeon's posture, or through the movement of the surgeon's hand, for example, the right hand.

[0058] Pushing, pulling, and twisting operations on such tubes can be determined to some extent by endoscopic images (internal images) and also by external images from camera 77. For example, distance detection and identification of fingers and elbows are possible using external images from camera 77. Finger and elbow identification is also possible using internal images from imaging unit 12a.

[0059] Furthermore, the operation of the dial can be performed in the left-right direction and the up-down direction of the curved section using the dial. Operation of the dial can be performed by the operator's hand, for example, the fingers of the left hand. Such operation of the dial can be determined to some extent using either the endoscopic image (internal image) or the external image from the camera 77. In addition, the up-down operation of the dial can also be determined using a gravity sensor.

[0060] In the following explanation, we show an example of determining the operation of the dial, i.e., the operation of the bending section, from the internal image. However, in some cases, it may be possible to electrically determine the operation of the dial in the endoscope system 12.

[0061] Furthermore, the external image captured by the external camera 13 allows for the determination of the operator's position and posture. Additionally, by capturing images of the patient's facial expression and posture with the external camera 13, the patient's facial expression and posture can also be determined.

[0062] Thus, in some cases, the same operation can be determined using either an internal or external image. The following explanation describes an example where the operation is determined using only one of the internal or external images, but the operation may also be determined using both external and internal images.

[0063] (Push-pull operation determination) Figures 4 to 6 are explanatory diagrams for explaining the operation determination regarding pushing and pulling of the insertion part.

[0064] Figure 4 shows images (external images) PE1 and PE2 captured by camera 77a, which filmed the operator 73 inserting the insertion device into the patient's body through the anus. In the example in Figure 4, operator 73 grasps the insertion device with his right hand 74R and holds the operating device 80 with his left hand 74L. External images PE1 and PE2 are images taken one after the other in time, and the change from external image PE1 to external image PE2 allows for the determination of the operator's pushing and pulling operations.

[0065] The dashed circles in Figure 4 indicate the elbow and fingers of the operator 73. Image changes in the elbow and fingers of external images PE1 and PE2 show that the operator 73 is moving his elbow from the outside to the inside of his body, bringing his right hand 74R, which is grasping the insertion device, closer to the patient's anus, thereby attempting to insert the insertion device into the operator's body through the anus. Due to this insertion operation, the distance from the edge of the screen to the tip of the operator 73's elbow is greater in external image PE2 than in external image PE1 (distance D1). In other words, distance D indicates information related to the insertion operation. The second time-series image analysis unit 22 can determine this distance D from the image changes.

[0066] The operator's movements 73 can also be observed by camera 77b, which takes images from the head side of the patient 71. Figure 5 shows external images PE3 and PE4, which were acquired in sync with external images PE1 and PE2 in Figure 4, respectively, at the same timing. Therefore, the operator's pushing and pulling operations can also be determined by the change from external image PE3 to external image PE4.

[0067] The dashed circles in Figure 5 indicate the elbow and fingers of the surgeon 73. A comparison of the external images PE3 and PE4 in Figure 5 shows that the surgeon 73 is moving his elbow from the outside to the inside of his body, bringing his right hand 74R, which is grasping the insertion part, closer to the patient's anus, thereby attempting to insert the insertion part into the body through the surgeon 73's anus.

[0068] The amount of pushing and pulling of the tube can be determined by comparing external images PE1 and PE2, or external images PE3 and PE4, to determine the amount of change in the position of the operator's 73 elbow and hand, for example, distance D in Figure 4. For example, by having the information acquisition unit 23 select an external image and using the analysis results of the second time-series image analysis unit 22, the time-series operation information generation unit 24 can determine the pushing and pulling operation on the insertion part and the amount of that operation (pushing and pulling amount). That is, in the example in Figure 4, the second time-series image analysis unit 22 uses the image portion of the elbow in the external image as an in-image feature portion and determines the change in the in-screen position of the in-image feature portion in order to quantify the operation of the operator 73.

[0069] Figure 6 illustrates the push-pull operation and push-pull amount obtained by the time-series operation information generation unit 24. The upper section of Figure 6 is a graph showing the movement of the surgeon's arm (particularly the elbow image in this case) for analyzing the surgeon's push-pull operation of the insertion part, with time on the horizontal axis and the distance from the left edge of the screen, which indicates the change in the position of the surgeon's elbow, on the vertical axis. The lower section of Figure 6 shows a series of external images PE5 to PE7 acquired in time series by the camera 77. In each of the external images PE5 to PE7, the three vertical lines indicate the horizontal positions on the screen: H1 (reference position of the elbow), H2 (position of the tip of the insertion part), and H3 (position of the tip of the insertion part). External image PE5 was obtained by imaging near timing t1, for example, and external images PE6 and PE7 were obtained by imaging at timings t2 and t3, respectively.

[0070] The graph used to analyze the operator's pushing and pulling movements of the insertion tube shows a wave-like result, as the insertion tube is tubular, and when the operator holds (pinches) it with their fingertips and inserts it into the subject's body from the tip, their fingers bump against parts of the subject's body. Therefore, the operator needs to move their grip from the tip of the tube towards the base (proximal end) of the tube, resulting in the operator inserting the tube and then returning their arm to its original position. Typically, the patient is lying down so that the examination is performed in a relaxed state, and the direction of tube insertion is generally horizontal on a screen where gravity is positioned vertically. By analyzing the movement of the operator's arm and elbow in the horizontal direction of the screen, it is possible to determine whether insertion or removal is occurring.

[0071] Furthermore, the length of the endoscope's insertion section can reach 2000 mm, making it prone to significant bending and flexing within the body and lumen. During endoscope insertion and withdrawal, even if the endoscope is inserted 100 mm at the hand, this does not necessarily mean the tip will be inserted 100 mm further inside. Moving the tip without hand manipulation is difficult, making hand manipulation crucial for physicians. Even if the movement of the imaging unit at the endoscope's insertion section is detected by image, it is not easy to determine the amount of insertion by the physician. Therefore, checking hand movements using images from an external camera is often more helpful for insertion and withdrawal than relying on changes in the endoscopic image.

[0072] External image PE5 shows an image taken around the time when the surgeon 73 began insertion. As the surgeon 73 extends his elbow forward, at timing t2, the elbow moves a distance D from, for example, the left edge of the screen. In this case, external image PE6 is obtained, in which the frames change chronologically as time progresses. As described above, by comparing external image PE5 and external image PE6, the distance D, that is, the amount of insertion by the surgeon 73, can be determined.

[0073] For measuring the amount of pushing and pulling, various arbitrary and appropriate positions within the screen can be selected, such as the four corners, four sides, or the center. However, since it is sufficient to detect elbow movement, stationary objects within the screen can also be used as a reference. Whether the elbow movement is toward the subject can be determined by whether the image of the grasped endoscope insertion part is to the left or right of the elbow, or by specifying the subject's position or determining the image of the area where the subject is located. Since the forward direction is the insertion direction, the operator's face position and orientation can also be used, allowing for detection in various ways. Furthermore, when setting up an external camera, a camera position is selected that captures the operator's arm movement and the subject within the field of view. At that time, the left or right side of the screen horizontally is detected, and it is determined whether the elbow is moving from the left or right periphery toward the center of the screen, and that direction can be used as the insertion direction.

[0074] In other words, an external camera captures video, the position of the operator's elbow within the resulting external image is determined, the repeated positional changes over time are monitored, and the change in the direction of the repeated elbow movement (left to right or right to left) is used to quantify the insertion distance. The numerical value used can be either a calculated value representing how far the endoscope actually entered the body, or simply a numerical value based on the change in the image. When using such numerical information to assist with endoscopic operations, the insertion speed may be classified and instructions such as "insert slowly" or "insert carefully." In addition, instructions may be given to stop insertion or to withdraw the endoscope, in which case it is sufficient to obtain information such as whether insertion is being attempted or whether the endoscope has been withdrawn.

[0075] For example, using the length of a doctor's arm as a reference, if a doctor with an upper arm length of 40 cm moves their elbow about 1 / 10 of the length of their upper arm on the screen, it becomes possible to quantify the amount of insertion and withdrawal. In this case, the quantified information can be used to give even more precise instructions, such as inserting 2 cm or inserting at a speed of 3 cm per second.

[0076] In this embodiment, it is possible to graph the operator's operation data (see Figure 12). When graphing, the insertion amount is plotted on the vertical axis in the positive direction and the withdrawal amount in the negative direction. However, these are not cumulative amounts, but are displayed as plus or minus zero when the procedure is stopped or the direction is changed, thus representing the insertion amount at each moment. This method of representing numerical values ​​is thought to make it easier for the operator to concentrate on the most recent operation at that moment. When the operator encounters difficulties during insertion, they are aware of what angle and twist to use, how far to insert, or move in the withdrawal direction, and it was thought that instructions indicating the operation at that moment would be effective.

[0077] Furthermore, to display the data in this way, the horizontal axis represents the operation time, and when the operator's upper arm does not move for a predetermined time (for example, 3 seconds), it is considered that there is neither pushing nor pulling, so the vertical axis returns to the origin. When pushing in the insertion direction, the change over time is displayed in the "pushing" direction. When pulling in the withdrawal direction, the change over time is displayed in the "pulling" direction. In other words, the second time-series image analysis unit 22 sets the horizontal position of the image portion in which the operator's arm features are detected to zero when no change is detected in the horizontal direction for a predetermined time in relation to the elapsed operation time, and outputs a numerical value corresponding to the amount of change in each direction when there is a change in the insertion or withdrawal direction. Of course, a graph of accumulated amounts that sequentially accumulates these insertion and withdrawal amounts may also be used.

[0078] The operator 73, upon reaching distance D, releases his grip on the insertor with his right hand to prevent it from being pulled out before it hits the subject's body, and slides his elbow back to its original position while sliding it toward himself. External image PE7 shows the external image in this case. This operation allows the operator 73 to push the insertor in again. Normally, the speed of the elbow when returning to its original position is faster than the speed of the elbow when pushing the insertor in. When advancing the insertor during insertion, care is taken as the insertor is moved inside the body (while observing and confirming its movement) (the wave period, here illustrated as the rise becoming slower). However, when changing the hand position to the base of the insertor (when changing grip), speed is required to prevent the insertor from coming out of the body and from moving too much inside the body, so the wave period becomes shorter. Figure 6 illustrates that the speed of the decline from the peak at distance D becomes faster. This change in speed can also be used to determine whether the insertor is being pushed in or withdrawn.

[0079] Conversely, during withdrawal, the insertion part is moved cautiously while observing the inside of the body (the wave period becomes longer). However, if this continues, the operator's arm will be fully extended, so it is necessary to change the position of the hand from the base of the insertion part to the tip. In this case, speed is required to prevent the tip of the endoscope from suddenly slipping out of the body during the change of grip, and to avoid moving the tip of the endoscope too much inside the body, so the wave period becomes shorter. As shown in Figure 6, the wave decreases slowly and rises quickly, so the graph shows characteristics opposite to those during insertion.

[0080] In other words, the second time-series image analysis unit 22 analyzes the time changes of the image feature portion in the time-series image obtained by an external camera that photographs the endoscope operator. When the endoscope operator inserts the endoscope into the subject's body, the system analyzes the shape of the wave based on the numerical increase and decrease of the wave waveform obtained by representing the horizontal change in the position of the image portion (image feature portion) in which the operator's arm features are detected on the screen as a graph with the horizontal axis being the time axis, and determines whether an insertion operation, an extraction operation, or neither is being performed. Such a determination can also be made by analyzing the change in numerical values ​​indicating, for example, the position of the elbow per unit time, even without graphing. The determination can be made whether the movement of the elbow is detected or the movement of the arm including the elbow is detected. In other words, the system determines whether an insertion operation, an extraction operation, or neither is being performed by analyzing the speed of the repeated horizontal changes in the position of the image portion in which the operator's arm features are detected on the screen.

[0081] The period from timing t1 to timing t4 in Figure 6 shows the insertion operation performed by the operator 73. From timing t4 onward, it shows the movement of the operator's elbow during the removal operation. The elbow's speed is relatively slow when removing the insertion part, and relatively fast when returning the elbow to its position before removal, resulting in a unique wave-like waveform with the characteristics described above.

[0082] Thus, the time-series operation information generation unit 24 can obtain information on the method of insertion operation, including the amount of insertion operation per unit time and its changes, and the method of withdrawal operation, including the amount of withdrawal operation per unit time and its changes, from the elbow movements of the operator 73 during insertion and withdrawal of the insertion part. In other words, this application includes an invention of a technique that determines whether an insertion operation is being performed, an withdrawal operation is being performed, or whether an insertion operation is being performed, or whether an insertion operation is being performed, or whether an withdrawal operation is being performed, based on the analysis of the shape of the wave (which occurs due to the time difference between rising and falling) based on the numerical rise and fall of the change in the wave waveform shown, for example when it is graphed, by determining the time change of the feature portion in the image of a time-series image of an endoscope operator and detecting the horizontal change in the position of the image portion showing the operator's arm (arm feature detected) when the endoscope operator inserts the endoscope into the body of the subject.

[0083] In Figure 6, a simplified example is shown in which insertion and removal operations are determined solely by the movement of the operator's 73 elbow. However, the system may also utilize other movements of the operator's 73 obtained by the camera 77 to determine insertion and removal operations. For example, the insertion device can be inserted by shifting the operator's 73's center of gravity, or by moving the operator's 73's fingers. The information acquisition unit 23 and the time-series operation information generation unit 24 may determine push-pull operations by obtaining the operator's 73's posture, wrist and finger movements, etc., from external images obtained by the camera 77. Furthermore, the information acquisition unit 23 and the time-series operation information generation unit 24 may also use the results of the internal image analysis by the first time-series image analysis unit 21 as a supplement to determine insertion and removal operations.

[0084] (Recording) The time-series operation information generation unit 24 outputs the generated push-pull operation information to the operation data recording unit 40. The operation data recording unit 40 is composed of a predetermined recording medium and records information regarding the push-pull operation of the insertion part (tube). The information regarding the push-pull operation is collected over time whether the operator is pushing or pulling the insertion part and the amount of push-pull. The time-series operation information generation unit 24 may also graph the information regarding the push-pull operation before recording it to the operation data recording unit 40. Since synchronization between the internal image and the external image is established, the time axis of the push-pull operation information corresponds to the time axis of the image frames of the internal image. Therefore, at the timing when an image of a specific part is captured as an internal image, information regarding the push-pull operation when inserting into that specific part is obtained.

[0085] Furthermore, in order to establish a correspondence between each image frame of the endoscopic images and various operations such as pushing and pulling, the operation data recording unit 40 is also supplied with and records the determination results of the observation area and internal image information from the first time-series image analysis unit 21. By recording the determination results of the observation area and internal image information in the operation data recording unit 40, it is possible to determine the content of operations at a specific area based on the information recorded in the operation data recording unit 40 even after the examination.

[0086] (Determination of rotation amount) Figure 7 is an explanatory diagram for explaining the determination of the rotation (torsion) operation of the insertion part.

[0087] Figure 7 shows an internal image of a specific location. In Figure 7, the shaded areas indicate relatively dark areas, and the roughly circular shaded areas indicate the deeper parts of the lumen. The dashed lines on the internal image in Figure 7 represent vertical and horizontal lines (hereinafter, this horizontal line is simply referred to as the horizontal line) passing through the center of the image.

[0088] The first time-series image analysis unit 21 performs image analysis of the internal image and detects feature points of the internal image PI1, which is an image of a predetermined specific area. For example, the first time-series image analysis unit 21 may detect the pointed shape of the internal image PI1 enclosed by the dashed circle as a feature point. Note that internal images PI2 to PI4 in Figure 7 were obtained by imaging the same specific area as internal image PI1.

[0089] The first time-series image analysis unit 21 may determine a feature point located in the center of the first quadrant of a predetermined frame X when detecting a feature point as a feature part within an image. The feature point of the internal image PI1 is located at the edge of the first quadrant. In this case, depending on the movement of the insertion unit (imaging unit), the feature point may move out of the field of view. For this reason, the first time-series image analysis unit 21 selects the internal image PI2 as frame X and detects a feature point located in the center of the first quadrant. At a predetermined timing, the angle between the horizontal line on the internal image PI2 and the feature point is θ0 degrees. The first time-series image analysis unit 21 determines the angle θ0 between the horizontal line and the feature point.

[0090] The first time-series image analysis unit 21 determines whether there are any frames that are temporally adjacent to and similar to the internal image PI2. If there are none, the first time-series image analysis unit 21 determines whether the analysis is complete or not. If the analysis is complete, it terminates the process; otherwise, it determines feature points for another frame X.

[0091] The first time-series image analysis unit 21, if it determines that there are temporally adjacent and similar frames, determines that the feature points are the same as the feature points of frame X and are located in the center of the Nth quadrant of frame X+m. Then, the first time-series image analysis unit 21 determines the angle θ1 between the horizontal line and the feature points for the internal image PI3, which is frame X+m.

[0092] Figure 7 shows that, relative to the internal image PI2, the imaging unit rotated in a plane parallel to the imaging surface, i.e., the insertion unit was twisted, resulting in a change in the angle between the horizontal line and the feature point in image PI3 to θ1 degree, and in image PI4 to a change in the angle between the horizontal line and the feature point to θ2 degrees. In other words, the direction and magnitude of the change in the angle between the horizontal line and the feature point correspond to the twisting direction and amount of twisting of the insertion unit. The first time-series image analysis unit 21 obtains the twisting direction and amount of twisting as analysis results based on the changes in the internal image. In other words, the time-series operation information generation unit 24 obtains the time difference of m frames and the information of θ1-θ2 as information on the twisting operation. For example, the first time-series image analysis unit 21 may determine that a clockwise rotation of the feature point is a leftward twist, and a counterclockwise rotation of the feature point is a rightward twist.

[0093] In this way, the first time-series image analysis unit 21 obtains information regarding the twisting operation every m frames. The amount of twisting operation is a maximum of 180 degrees in both the left and right directions.

[0094] The information acquisition unit 23 and the time-series operation information generation unit 24 generate information related to twisting operations using the analysis results of the internal image by the first time-series image analysis unit 21 and output it to the operation data recording unit 40. The operation data recording unit 40 records information related to the twisting operation of the insertion part (tube). The information related to twisting operations is collected at time intervals corresponding to the time of the internal image, indicating in which direction the operator is twisting the insertion part and the amount of twisting. The time-series operation information generation unit 24 may also graph the information on twisting operations before recording it to the operation data recording unit 40. In this way, information related to the twisting operation when inserting into a specific part is obtained at the timing when an image of that specific part is captured as an internal image.

[0095] (Bending Determination) Figure 8 is an explanatory diagram for explaining the determination of the amount of bending during the bending operation of the insertion part.

[0096] Figure 8 shows an internal image of a predetermined specific region using the same notation as in Figure 7. The first time-series image analysis unit 21 performs image analysis of the internal image and detects feature points of the internal image PI11, which is an image of a predetermined specific region. For example, the first time-series image analysis unit 21 may detect the portion enclosed by the dashed circle in the internal image PI11 as feature point lm0. Note that internal images PI12 and PI13 in Figure 8 were obtained by imaging the same specific region as internal image PI11. When detecting feature points, the first time-series image analysis unit 21 may determine feature points that are located in the central part of the internal image PI11, which is an image of a predetermined frame X. In the example in Figure 8, feature point lm0 is an image portion that shows approximately the center in the depth direction of the lumen.

[0097] The first time-series image analysis unit 21 determines whether there are any frames that are temporally adjacent to and similar to the internal image PI 12. If there are none, the first time-series image analysis unit 21 determines whether the analysis is complete or not. If the analysis is complete, it terminates the process; otherwise, it determines feature points for another frame X.

[0098] The first time-series image analysis unit 21, if it determines that there are temporally adjacent and similar frames, determines the position of the feature point lm1 of frame X + m, which is the same feature point as the feature point of frame X. Then, the first time-series image analysis unit 21 determines the time difference, direction of position change, and amount of change for the m frames of the internal image PI12, which is frame X + m.

[0099] In Figure 8, by changing the orientation of the optical axis of the imaging unit up, down, left, and right, that is, by operating the dial provided on the operation unit 80, the curved part of the insertion unit is curved left, right, up, and down. As a result, with respect to the internal image PI 11, image PI 12 shows that the position of the feature points in the screen has changed horizontally, and image PI 13 shows that the position of the feature points in the screen has changed vertically. In other words, the direction and magnitude of the change in the horizontal position of the feature points correspond to the direction and magnitude of the left-right orientation of the curved part, i.e., the operation of the left-right dial, and the direction and magnitude of the change in the vertical position of the feature points correspond to the direction and magnitude of the up-down orientation of the curved part, i.e., the operation of the up-down dial.

[0100] The first time-series image analysis unit 21 obtains the dial operation direction and operation amount as analysis results based on the changes in the internal image. The time-series operation information generation unit 24 obtains the time difference of m frames and the dial operation direction and operation amount as dial operation information. In the example in Figure 8, internal image PI12 is an image obtained by operating the dial to the right from the state of internal image PI11, and internal image PI13 is an image obtained by operating the dial downward from the state of internal image PI11.

[0101] In this way, the first time-series image analysis unit 21 obtains information regarding dial operation for every m frames. The upper limits of the dial operation amount may be, for example, 210 degrees up, 90 degrees down, and 100 degrees left and right.

[0102] The information acquisition unit 23 and the time-series operation information generation unit 24 generate information related to dial operation using the analysis results of the internal image by the first time-series image analysis unit 21 and output it to the operation data recording unit 40. The operation data recording unit 40 records information related to the dial operation. The information related to dial operation is collected for each time period corresponding to the time of the internal image, indicating which direction the operator is rotating the dial and the amount of rotation. The time-series operation information generation unit 24 may also graph the dial operation information before recording it to the operation data recording unit 40. In this way, information related to the dial operation when inserting into a specific area is obtained at the timing when an image of that specific area is captured as an internal image.

[0103] In this way, the data processing device 20 quantifies the characteristics of the operations performed by the endoscope operator when inserting or withdrawing an endoscope with an imaging unit at its tip into the body of a subject and acquires operation data. Specifically, a first time-series image analysis unit for determining the time changes of image features in time-series images obtained by the endoscope imaging unit and a second time-series image analysis unit 22 for determining the time changes of image features in time-series images obtained by an external camera that photographs the endoscope operator analyze the types of operations performed by the endoscope operator into categories such as endoscope insertion / withdrawal, twisting of the endoscope insertion unit, and angle of the endoscope tip, and the time-series operation information generation unit 24 analyzes the changes in the amount of operation for each category to generate operation data.

[0104] Furthermore, if the information on the amount of operations acquired by the time-series operation information generation unit 24 is displayed on the same time axis, it is possible to create a graph like the one in Figure 12. However, it is preferable to provide a synchronization unit that synchronizes the time-series images obtained by the endoscope imaging unit with the time-series images obtained by the external camera. Based on the time-series changes of each synchronized image, the type of operation performed by the endoscope operator can be divided into items such as endoscope insertion and withdrawal, twisting of the endoscope insertion part, and angle of the endoscope tip, and the change in the amount of operation can be analyzed for each item. By combining this with the endoscope image (internal image) at each timing, it is possible to determine under what circumstances (situation in which the endoscope image was obtained) and by what operation insertion into the lumen can be performed, and guide information can be obtained. As a presentation of this guide information, simply displaying the graphed image in Figure 12 is sufficient to convey the content to the user. Furthermore, the content that can be read from such a graph may be converted into text or illustrated by the guide display generation unit 63. Just as body shapes differ among subjects, the structure within the lumen also varies. However, similar images can often be obtained of the lumen, and it can be difficult to determine a difficult insertion situation from just one image or frame. Therefore, it is advisable to determine a difficult endoscopic insertion situation by using multiple images obtained in chronological order.

[0105] In this way, the control unit 60 acquires time-dependent change information between the operator's operation information for a specific case and the endoscopic image obtained by the imaging unit 12a provided at the tip of the endoscope, and records the operation information, endoscopic image, and change information in association with the information from the operation data recording unit 40. Based on this information, the control unit 60 references the operation data of the endoscope when the endoscopic image of the specific case at a first timing in each case changes to the endoscopic image at a second timing in the same case. Based on the endoscopic operation data, it generates multiple guide information and enables guidance based on the generated guide information.

[0106] Furthermore, the time-series operation information generation unit 24 also records information regarding the operation of switches provided on the endoscope operation unit in the operation data recording unit 40. If the switch is an on / off switch, the time-series operation information generation unit 24 may indicate that an on or off operation was performed by the pulse shape. In addition, by applying annotation using the scale provided on the insertion unit, the amount of insertion of the insertion unit into the body can also be inferred.

[0107] The system should employ multiple switches, not just one, with each operation independently distinguishable. These switches correspond to functions such as water supply, air supply, suction, light source switching, special light observation, and image processing. Endoscopes may be inserted, observed, or diagnosed using these switches. Recording the situations in which a skilled model physician performs endoscopic operations (including the relationship between endoscopic images and switch operations) can provide valuable reference for less experienced physicians. For example, when anterior obstruction occurs during endoscope insertion, air may be injected into the lumen to inflate it. Recording the timing of this switch operation, along with the change in the endoscopic image from a contracted lumen to an expanded state, allows for the determination of the techniques used by the skilled model physician.

[0108] In this way, the operation data recording unit 40 records various operation data related to at least one endoscopic operation, such as pushing and pulling the tube (insertion part), twisting, dial operation, and switch operation, which are the first and second operation information, in a way that can be distinguished by operation unit. These operation data may also be recorded in graph form. Multiple pieces of information for each examination are recorded in the operation data recording unit 40. Specifically, the operation data recording unit 40 records endoscopic images and operation information as operation data, and also records site information indicating which part the operation data pertains to, and the UPD image (UPD information) recorded when the operation data was obtained. The operation data recording unit 40 also records the subject profile, operator profile, and examination name (part) for each examination. Furthermore, as will be described later, the operation data recording unit 40 also records information on failed operations by frame and part obtained in the event of a failed operation.

[0109] Furthermore, there are typical scenarios in endoscope insertion procedures where insertion becomes difficult. Therefore, the operation data recording unit 40 records a database of internal images and operation data for such typical scenes, which are designated as typical scenes. These internal images and operation data for typical scenes can be acquired by the guide information generation system 10, and for typical scenes, they are recorded in the typical scene recording unit 41 of the operation data recording unit 40. Model operations in typical scenes provide recovery time-series images (internal images) and their operation data that overcome the insertion difficulty situation.

[0110] The guide information generation unit 25 converts the operation data recorded in the operation data recording unit 40 into guide information. The guide information generation unit 25 obtains guide information for each operation unit. This guide information is supplied to the control unit 60.

[0111] (Endoscope Operation Practice Unit) In Figure 1, the endoscope operation practice unit 50 includes a simulated operation unit 51 and a data processing unit 55. The simulated operation unit 51 includes a right-hand insertion unit operation determination unit 52, a left-hand insertion unit operation determination unit 53, and a synchronization unit 54. The right-hand insertion unit operation determination unit 52 includes an insertion / removal operation determination unit 52a and a twisting operation determination unit 52b. The left-hand insertion unit operation determination unit 53 also includes an up / down bending operation determination unit 53a, a left / right bending operation determination unit 53b, and a switch (SW) operation determination unit 53c.

[0112] Figure 9 is an explanatory diagram illustrating an example of the configuration of the simulated operation unit 51. The simulated operation unit 51 has simulated operating devices 57 and 59 for simulating operations on the endoscope insertion section. The simulated operating device 57 is made of a tube made of the same material as the endoscope insertion section, has the same flexibility as the insertion section, and has the same diameter as the insertion section, and is operated, for example, with the right hand 74sR. The simulated operating device 57 does not have to be in the shape shown in Figure 9, but a long tube would be difficult to handle, so it is made in a loop shape. The insertion / removal operation determination unit 52a and the twisting operation determination unit 52b of the right-hand insertion section operation determination unit 52 are made of a sensor unit 58. For example, a predetermined pattern is formed on the surface of the simulated operating device 57, and by optically detecting this pattern with the sensor unit 58, the amount of pushing / pulling and twisting (torsion) of the simulated operating device 57 can be detected.

[0113] Furthermore, the vertical bending operation determination unit 53a and the horizontal bending operation determination unit 53b of the left-hand insertion section operation determination unit 53 are composed of a simulated operating device 59. The simulated operating device 59 has a configuration similar to, for example, a dial provided on the operating section of an endoscope, and is operated, for example, with the left hand 74sL. The simulated operating device 59 includes a dial 59a for vertical bending operation and a dial 59b for horizontal bending operation. The vertical bending operation determination unit 53a of the left-hand insertion section operation determination unit 53 determines the amount of rotation of dial 59a, and the horizontal bending operation determination unit 53b of the left-hand insertion section operation determination unit 53 determines the amount of rotation of dial 59b.

[0114] The simulated operation unit 51 has a simulated switch (not shown) as a simulated operating device, and the SW operation determination unit 53c of the left hand insertion unit operation determination unit 53 is capable of determining the operation of this simulated switch. The simulated switch is a simulation of switches for operating various medical devices (not shown), such as air supply and water supply devices.

[0115] The synchronization unit 54 synchronizes the determination results of the right-hand insertion unit operation determination unit 52 and the left-hand insertion unit operation determination unit 53. The determination results of the right-hand insertion unit operation determination unit 52 and the left-hand insertion unit operation determination unit 53 are output to the data processing device 55. The data processing device 55, which acts as a second data acquisition device, has the same configuration as the data processing device 20 and acquires operation data by quantifying the pushing and pulling of the pipe, twisting, and up, down, left, and right operations on the dial based on user operations on the simulated operation unit 51.

[0116] Furthermore, the data processing device 55 acquires time-dependent change information between the operator's operation information and the endoscopic image, enabling it to associate the operation information, endoscopic image, and change information for determination. It is also possible to associate each frame of the endoscopic image with the acquisition timing.

[0117] The control unit 60 includes a selection unit 61, an operation determination unit 62, a guide display generation unit 63, a display control unit 64, and an input unit 65. The selection unit 61 of the control unit 60 is given a case selection signal, which will be described later, and the selection unit 61 reads operation data for the operation unit of the corresponding part of the case corresponding to the case selection signal from the operation data recording unit 40. The operation determination unit 62, which acts as a guide generation unit, determines whether the operation on the simulated operation unit 51 is appropriate by comparing the operation data from the data processing device 55 of the endoscope operation practice unit 50 with the operation data from the operation data recording unit 40.

[0118] The guide display generation unit 63, which functions as a guide generation unit or guide presentation unit, creates operation data for operation units from the operation data recording unit 40, that is, display data for guide displays based on guide information. In this way, the guide display generation unit 63 makes it possible to provide information based on guide information when the simulated operating device is operated.

[0119] Furthermore, the guide display generation unit 63 creates display data for the guide display based on the operation data from the data processing device 55 of the endoscope operation practice unit 50. The guide display generation unit 63 also generates display data for the guide display based on the judgment result of the operation judgment unit 62. In other words, in this embodiment, as described above, not only can guide information indicating the operation content of each operation unit be presented based on the operation data recorded in the operation data recording unit 40, but the operation judgment unit 62 and the guide display generation unit 63 can also compare the operation data from the operation data recording unit 40 with the operation data from the simulated operation unit 51 for a predetermined operation unit, and generate and present guide information based on the comparison result.

[0120] The display control unit 64 provides the guide display generated by the guide display generation unit 63 to the display unit 66. The display unit 66 is composed of a display device such as an LCD (liquid crystal panel), and is controlled by the display control unit 64 to display the guide display as a guide screen on the display screen. The display control unit 64 can also display various menus on the display unit 66. Alternatively, the display unit 66 may be composed of smart glasses, and the guide display may be virtually displayed on the smart glasses.

[0121] While the term "guide display" was used to illustrate a clear example of displaying a guide on a screen, it is also possible to implement tactile assistance using voice assistance or vibration. In other words, the guide display generation unit could also be referred to as the guide generation unit.

[0122] The input unit 65 can receive user input operations, and the control unit 60 operates in accordance with the operations received by the input unit 65. For example, the input unit 65 can be configured as a touch panel provided on the display unit 66, and the operation of each part of the control unit 60 is determined according to the user's input operations, such as touch operations on the menu displayed on the display unit 66.

[0123] For example, the input unit 65 can generate a case selection signal based on user operation to identify which case's operation data to use in order to use guide information suitable for insertion practice. The input unit 65 as a case selection unit may include not only a touch panel but also an input operation unit such as a keyboard (not shown). The case selection signal identifies the operation data to be used for insertion practice and includes information that specifies, for example, the target site of operation, the age and gender of the patient (subject), and information that identifies the operator (the endoscopy operator who performed the model operation). Note that the functions of such an input unit 65 may also be provided in the endoscopy operation practice unit 50. The input unit 65 allows the operation of the endoscopy operator to be identified and operation data corresponding to the identified operation to be selected before operating the simulated operation device.

[0124] Furthermore, if the operation determination unit 62 of the control unit 60 determines, based on the results of the insertion practice, that the operation to the simulated operation unit 51 is inappropriate, i.e., that the operation is a failure, it provides the operation data of the operation to the operation data recording unit 40 for recording. The control unit 60 may also count the number of failed operations for each body part to be inspected, determine whether the number of failed operations exceeds a predetermined threshold, and display a guide display based on the operation data of the failed operations where the number of failed operations exceeds the predetermined threshold on the display unit 66 at the start of practice for that body part.

[0125] (Operation) Next, the operation of the embodiment configured as described above will be explained with reference to Figures 10 to 15. Figures 10 to 12 are for explaining the generation of guide information by the guide information generation system 10, with Figure 10 being a flowchart and Figures 11 and 12 being explanatory diagrams.

[0126] (Guide Information Generation) Figure 11 shows internal images PI21-PI26 and external images PE21-PE26 obtained by the guide information generation system 10 during the process of endoscope insertion by a skilled insertion operator. In Figure 11, the shaded areas in internal images PI21-PI26 represent the image portion deep within the lumen.

[0127] Internal images PI21 and PI22 include the image portion at the back of the lumen. Therefore, at the timing of these image frames, the operator did not miss the back of the lumen. However, while internal image PI21 shows the back of the lumen in the image, at the timing of internal image PI22 the back of the lumen is located at the edge of the image. As a result of further pushing the insertion part in this state, internal image PI23 shows an elliptical bubble, but the image portion at the back of the lumen is not included. In other words, internal images PI23 to PI25 indicate that the lumen to be inserted has been lost.

[0128] An experienced surgeon, in internal image PI23, attempts to push the insertion tube further in, but judging that this insertion operation will not be successful, calmly withdraws the insertion tube (internal image PI24) and attempts to insert it again (internal image PI25). As a result, as shown in internal image PI26, the image portion at the back of the lumen reappears in the image. From this point onward, the surgeon inserts the insertion tube towards the back of the lumen.

[0129] In this series of operations, the data processing device 20 acquires operation data and records the acquired operation data in the operation data recording unit 40. In the example in Figure 11, the operations of pushing, pulling, and pushing are performed, and these can be considered as three operation units: insertion, insertion failure, and recovery operation leading to successful insertion.

[0130] Figure 10 shows an example of a method for creating guide information during an examination performed by a skilled endoscope operator. Internal and external images acquired by the endoscope operation monitoring system 11 are supplied to the data processing device 20, and operation data is acquired. The operation data is provided to the operation data recording unit 40. In addition, information such as the operator, subject profile, and examination details are acquired by the target unit classification unit 30 and the basic information acquisition unit 31, and provided to the operation data recording unit 40. This data is tagged and recorded by the operation data recording unit 40 (S1 in Figure 10).

[0131] The guide information generation unit 25 of the data processing device 20 selects an image with good image quality from the acquired frame group, for example, at the start of insertion, and sets it as the first image (S2, S3). The guide information generation unit 25 advances to the frame after the first image (S4) and determines whether there has been a change in the state of the "continuous operation performed by the same action" that has been performed continuously since the first image. The guide information generation unit 25 considers the period during which this continuous operation is performed as an operation unit, and if there is no change (NO in S5), it continues to advance the frames (S4). If a change occurs, the guide information generation unit 25 determines that the operation unit has ended and the next operation unit has begun (YES in S5), and converts this continuous operation into guide information as an operation unit (S6). Furthermore, the image before the change in the continuous operation, i.e., the last image of the operation unit, is designated as the second image, and the first image and the second image are associated with the guide information of the operation unit (S7), and recorded in the operation data recording unit 40.

[0132] Furthermore, the guide information generation unit 25 associates the part name and UPD image with the operation data of the operation unit and packages this data (S8). As a result, when reading the operation data of the operation unit, the first image, second image, part name, UPD image, etc. are read simultaneously with the operation data of the operation unit.

[0133] The guide information generation unit 25 determines whether the inspection is complete or not (S9). If the inspection is not complete, the guide information generation unit 25 returns to processing S3 and repeats S3 to S8. If the inspection is complete, it terminates processing.

[0134] Figure 12 illustrates the operation data for each operation unit. Figure 12 shows that the insertion part 79 is pushed and pulled and twisted with the right hand 74R, and the dials 83a and 83b (hereinafter referred to as dial 83 when neither is distinguished) for up-down and left-right bending operations are operated with the left hand 74L to perform the insertion operation. As a result, the time-series operation information generation unit 24 obtains graphed operation data.

[0135] The guide information generation unit 25 detects the operation unit and sets the internal image PI21 as the first image and the internal image PI22 as the second image. Within the interval of the operation unit, the tube is pushed and twisted to the left, and at the same time, the rotation of the left / right dial causes the insertion part to change direction to the right. Figure 12 shows the UPD image at the timing when the first image was obtained and the UPD image at the timing when the second image was obtained. The area within the dashed lines in Figure 12 can be displayed on the display screen of the display unit 66 by the guide display generation unit 63 and the display control unit 64 of the control unit 60.

[0136] In the example shown in Figure 12, the period from timing t0 to t1 in the graphed operation data corresponds to the operation unit in which the first and second images are set. By performing the operations shown in the graph of Figure 12 from the timing of the first image, the second image is obtained.

[0137] Furthermore, as shown in Figure 11, if a series of operations involving withdrawing and reinserting is suitable for repeated practice of the operation, this series of operations may be set as an operation unit. In this case, the guide information generation unit 25 may, for example, treat the recovery operation from internal image PI22 to internal image PI26 as a single operation unit. In this case, the guide information generation unit 25 sets internal image PI22 as the first image and internal image PI26 as the second image.

[0138] (Insertion Practice) Figures 13 to 15 are for explaining insertion practice; Figure 13 is a flowchart, and Figures 14 and 15 are explanatory diagrams.

[0139] Figure 14, using the same notation as Figure 12, is intended to illustrate the comparison between the operation data acquired by the simulated operation unit 51 (hereinafter referred to as simulated operation data) and the operation data recorded in the operation data recording unit 40 (hereinafter referred to as model operation data). Note that the area within the dashed lines in Figure 14 may be displayed on the display unit 66.

[0140] Figure 14 shows that the operator uses their right hand 74sR to perform push, pull, and twist operations on a simulated operating device 57 that simulates the insertion part, and their left hand 74sL to operate a simulated operating device 59 that simulates a dial for bending the curved part in the up, down, left, and right directions. This simulated operation allows the data processing device 55 to obtain operation data (simulated operation data). In Figure 14, in the graph showing the operation data, thin lines represent model operation data and thick lines represent simulated operation data.

[0141] Furthermore, this simulated operation is performed on the premise that it starts from the situation in which the first image is acquired and ends in units of one operation in the simulated operation data. In the example in Figure 14, the first image and the UPD image obtained when the first image is acquired, as well as the second image and the UPD image obtained when the second image is acquired, are also displayed. As is clear from the comparison between the graphed model operation data and the simulated operation data, among the operations of pushing and pulling the pipe, twisting, and dialing, the twisting operation in particular differs significantly between the model operation data and the simulated operation data. The × mark on the second image indicates that the simulated operation data obtained as a result of the simulated operation does not match the model operation data, and therefore the second image cannot be obtained by the simulated operation, i.e., the simulated operation is a failed operation.

[0142] In this way, the success or failure of an operation is immediately apparent, allowing for repeated practice to approximate the operation of a physician (model operation) recorded in the database (operation data recording unit 40). Furthermore, this embodiment, while guiding the user on what operation to perform next while viewing an endoscopic image of a specific organ in a specific case, is also a technology that guides the user on what operation to perform following that image. Therefore, it is not limited to an invention in the field of practice devices or practice methods, but can also be described as an invention of an endoscopic operation guidance method that, for a specific case, acquires time-dependent change information between the operator's operation information and the endoscopic image obtained by the imaging unit 12a provided at the tip of the endoscope, and generates operation guide information by referencing the operation data to be performed following the endoscopic image of the specific case at a first timing from the operation data recording unit 40, which records the operation information, endoscopic image, and change information in association.

[0143] Furthermore, even before acquiring simulated operation data, a guide showing predicted failures based on past performance and the operator's profile may be displayed to warn users.

[0144] The model operation data recorded in the operation data recording unit 40 can be thought of as a numerical representation of the characteristics of an operation performed by a skilled physician when inserting, examining, or withdrawing an endoscope into the body of a specific subject, using the endoscopic operation quantification system, which is one of the features of this embodiment. In other words, the skilled physician's operation is divided into items such as endoscope insertion / withdrawal, twisting of the endoscope insertion section, and angle of the endoscope tip, and the presence or absence of the operation and the change in the amount of the operation are analyzed and generated for each item, so the results of this analysis become the standard model operation data. The simulated operation data generated during simulated operation by an unskilled person is also obtained by dividing the type of operation into items such as endoscope insertion / withdrawal, twisting of the endoscope insertion section, and angle of the endoscope tip, and determining the presence or absence of the operation and the change in the amount of the operation over time for each item. Guide information may be obtained by comparing the model operation data and the simulated operation data at each timing in the time series and combining them with the images obtained by the endoscopic imaging unit.

[0145] In this way, by comparing the model operation data recorded in the operation data recording unit 40 by skilled physicians with the simulated operation data obtained through simulated operation using the simulated operation unit 51, it is possible to determine whether the simulated operation matches the operation (model operation) of a skilled physician. Furthermore, by using the simulated operation unit 51, the operator can easily repeat the practice until the simulated operation matches the model operation, while confirming whether it matches. In addition, by generating guide information for correct operation based on the model operation data and conveying it to the operator performing the simulated operation, insertion practice can be made even more effective.

[0146] During such insertion practice, the user, such as a doctor, decides which body part they want to practice on. The selection unit 61 of the control unit 60 performs the selection process for the subject and the doctor in S11 of Figure 13. That is, the selection unit 61 controls the display control unit 64 to display the menu display Ds1 shown in Figure 15 on the display screen of the display unit 66. The menu display Ds1 is for selecting the examination subject to be practiced. This procedure, together with S11 and S12, corresponds to a selection step, but it may be possible to select from more finely classified options, and a selection method that combines automatic, semi-automatic, and recommendation may be adopted to reflect the user's skills and needs. By selecting the examination target body part column with a touch operation of the finger 91, body parts such as the large intestine and stomach can be selected. Figure 15 shows that the user has specified "large intestine". In addition, the gender and age of the patient can be specified by touching the "gender" and "age" columns. Furthermore, by touching the "doctor" column, the doctor who performed the examination on the patient can be identified. Then, by selecting the "Candidate" column from the multiple candidates obtained through the above operations, it is possible to select a specific candidate.

[0147] The selection unit 61 accesses the operation data recording unit 40, reads the UPD image of the area selected by the user operation, and provides it to the display control unit 64 to display the menu display Ds2 (S12). The menu display Ds2 is for specifying the insertion practice start position and includes the UPD image PU1 read from the operation data recording unit 40. The user specifies which part of the large intestine they want to start insertion practice from by specifying a specific position of the UPD image PU1 with a touch operation of their finger 91. Such specification is possible because the data corresponding to the first timing of starting practice is organized in the case-specific database recorded in the operation data recording unit 40 in Figure 1, which contains data indicating when and to which part was reached and how the endoscopic image was recorded at that time. In other words, in this embodiment, by specifying the part of the target organ in a particular case, it is possible to refer to or read information such as which frame of the corresponding image data corresponds to that, and what the subsequent operation was.

[0148] If an UPD image is not available, the system may allow the user to specify the starting body part for practice by having them input the name of the body part or by displaying a model image stored in memory (not shown). Furthermore, there are inference models that can determine body parts from images, making it possible to determine the relationship between which body part corresponds to which image frame was captured at which time.

[0149] The selection unit 61 reads model operation data and internal images corresponding to the part specified by the user, i.e., the first image, the second image, and the model operation data for the operation unit, and provides them to the display control unit 64 to display the guide screen Dg1 (S13). The guide screen Dg1 includes an internal image PI31, which is the first image, corresponding to the position specified by the user. If an UPD image corresponding to the timing of the first image exists in the operation data recording unit 40, that UPD image may also be included in the guide screen Dg1.

[0150] The selection unit 61 determines whether the number of failed operations at the location (or first image) specified by the user is greater than a predetermined number (S14). If the number of failed operations is greater than a predetermined number (S14, YES), the selection unit 61 may read the simulated operation data of the failed operations from the operation data recording unit 40 and display the operation examples based on the simulated operation data of the failed operations on the display unit 66 (S15).

[0151] Furthermore, the guide display generation unit 63 may control the display control unit 64 to include a guide display Dgd1 on the guide screen Dg1, along with instructions for the physician's operation recorded in the operation data recording unit 40 (S16). The guide display Dgd1 is generated based on the model operation data of the operation unit. For example, in order to insert the insertion unit from the state of the first image to the state in which the second image is obtained, the guide display generation unit 63 may display instructions explaining the operation, such as "Insert the dial by twisting it slowly to the left while operating it to the right." The guide display generation unit 63 may also output content corresponding to such a guide display Dgd1 as sound through a speaker (not shown).

[0152] Thus, in S16, various types of guides may be made selectable. For example, if multiple guide types are displayed as thumbnails in the guide display Dgd1 in Figure 15 and made selectable, the user can choose their preferred guide. Examples of each of the multiple guide types are shown in Figures 16 and 17.

[0153] These guide formats are created based on the original operation data (a record of temporal changes broken down into multiple operation elements), taking into account the type, direction, and quantity of the operation. The content of this guide can be conveyed to the user by verbalizing the operation of the control unit.

[0154] Furthermore, if the simulated operation unit 51 employs a sensor unit 58A and a simulated operation device 59A, which have actuators added to the sensor unit 58 and simulated operation device 59, respectively, the guide display generation unit 63 may provide operation guidance by driving these sensor unit 58A and simulated operation device 59A. For example, the guide display generation unit 63 may drive the sensor unit 58A to twist the simulated operation device 57 to the left and rotate the left / right bending dial 59Ab to the right, thereby providing guidance using the user's sense of touch. Note that dial 59Aa is a dial for up / down bending.

[0155] Next, the user operates the simulated operation unit 51 to practice insertion. Simulated operation data based on the user's operation of the simulated operation unit 51 is acquired by the data processing device 55 and supplied to the control unit 60. The operation determination unit 62 of the control unit 60 determines whether the user's simulated operation data (operation detection result) matches the physician's model operation data recorded in the operation data recording unit 40 (S17). For example, the operation determination unit 62 may determine that the simulated operation data matches the model operation data if the ratio of differences between the simulated operation data and the model operation data for each operation data of push-pull operation, twist operation, and dial operation is smaller than a predetermined threshold for all operation data.

[0156] The operation determination unit 62 displays a guide screen Dg2 or Dg3 according to the operation determination result. If the simulated operation data obtained by the user's operation of the simulated operation unit 51 matches the model operation data read from the operation data recording unit 40, the operation determination unit 62 determines that the insertion is successful and displays the endoscopic image that should be obtained after successful insertion, i.e., the second image, the internal image PI32, and also displays a circle mark Ps indicating that the insertion was successful (S22). The control unit 60 may also display an UPD image on the guide screen Dg2.

[0157] Furthermore, if the operation determination unit 62 does not determine that the simulated operation data and the model operation data match, the operation determination unit 62 records the simulated operation data in a memory not shown (S18). Next, the operation determination unit 62 determines whether the detection result of the simulated operation differs significantly from the model operation. For example, the operation determination unit 62 may determine the ratio of the difference between the simulated operation data and the model operation data for each operation data of push-pull operation, twist operation, and dial operation, and if there is an operation where the ratio deviates significantly by more than a predetermined threshold, it may determine that the simulated operation differs significantly from the model operation. If the simulated operation and the model operation do not differ significantly (NO determination in S19), the operation determination unit 62 may return to processing in S16 and display the guide display Dgd1.

[0158] On the other hand, if the simulated operation and the model operation differ significantly (YES judgment in S19), the operation determination unit 62 provides the simulated operation data to the operation data recording unit 40 for recording (S20). Such simulated operation data can be used to construct an inference model for operation support. In such a simulated operation, assuming that the insertion fails, the operation determination unit 62 displays an "x" mark Pf indicating insertion failure on the internal image PI32, which is the second image to be obtained after successful insertion, using the guide screen Dg3. In this case, the operation determination unit 62 may also display a message recommending retrying using the guide display Dgd2 (S21). The guide display Dgd2 of the operation determination unit 62 may also show the difference between the simulated operation and the model operation.

[0159] In this embodiment, images from the endoscope and an external camera are captured in sync, and information regarding endoscopic operation (model operation data) is obtained by analyzing each image. The model operation data is then divided into predetermined operation units, and guide information is generated for each operation unit. Operation data (simulated operation data) is also obtained for simulated operations performed using a simulated operating device. By comparing the model operation data and simulated operation data for each operation unit, it is possible to present guide information for operation units that are effective for repetitive practice, and an extremely high level of practice effectiveness can be obtained for insertion practice.

[0160] (Second Embodiment) Figures 16 to 18 are explanatory diagrams showing a second embodiment. The hardware configuration of this embodiment is the same as that of the first embodiment. This embodiment shows another example of guide display. Figures 16 to 18 show, for example, what is displayed on the display screen of the display unit 66, but guides may also be provided by voice using a speaker (not shown). In addition, various guide screens may be displayed on smart glasses.

[0161] The guide screen Dg11 in Figure 16 displays the internal image PI41, which is the first image read from the operation data recording unit 40, on the right side of the screen, and the guide display Dgd11 on the left side of the screen. The guide display generation unit 63 displays the simulated operation data as a graph on the graphed operation data of the model operation data as the guide display Dgd11 in real time. In the guide display Dgd11, thin lines indicate the model operation data and thick lines indicate the simulated operation data. By comparing the operation data, the operator performing the simulated operation can notice the differences in the model operation data and practice while correcting the operation.

[0162] The guide screen Dg12 in Figure 16 displays the internal image PI42, which is the first image read from the operation data recording unit 40, on the right side of the screen, and the guide display Dgd12 on the left side of the screen. The guide display generation unit 63 displays the movement of the simulated operation unit 51 based on the model operation data as an animation in accordance with the simulated operation, as the guide display Dgd12. The operator performing the simulated operation can recognize the operation based on the model operation data by using the guide display Dgd12.

[0163] Furthermore, to make practice feel like a game, case studies and body part information can be categorized into difficulty levels, and the system can display how far the same operator has been able to operate correctly, or make the operator aware of their appropriate skill level, allowing them to compare themselves to other operators, record their history, and feel a sense of progress. In addition, the system can recommend the next case study according to the skill level. Since many Japanese people undergo examinations at a specific age, cases for that age group may be recommended preferentially, even if they are difficult. In this embodiment, there is a selection step to select a specific case study, but this selection step may be performed automatically or semi-automatically based on the history of success or failure of past simulated operations, or control such as displaying recommendations for cases and body parts based on the history of success or failure of past simulated operations and difficulty information may be implemented. Such control can be rephrased as using profile information linked to the operator. In other words, in this embodiment, to select a specific case study, the control unit 60 or the like can be provided with steps such as selecting profile information of the case subject and the case operator, or inputting the profile information of the operator using this operation guide.

[0164] The guide screen Dg13 in Figure 16 displays the internal image PI43, which is the first image read from the operation data recording unit 40, on the right side of the screen, and the guide display Dgd13 on the left side of the screen. The guide display generation unit 63 displays a doctor avatar that moves according to the model operation data as the guide display Dgd13.

[0165] Such avatar displays are possible by using motion capture technology to digitize the doctor's movements based on images captured by an external camera. Motion capture records human "movement" by recording the position and movement of characteristic parts of the human body, such as joints. For example, marks called "markers" can be placed on joints, and their position and movement can be measured in three dimensions and recorded as data. Image features can also be used as a substitute for markers. The movement of each joint can be mapped to the movement of the avatar's joints to illustrate posture and other aspects. Alternatively, images can be created using generative AI technology. The doctor's avatar's movements can be reproduced and displayed in real time according to the model operation data. Furthermore, the avatar's movements can be further reinforced by training the doctor's avatar with various case examples using the information from the operation data recording unit (database) 40 in Figure 1. The avatar may be displayed as a 3D image in addition to a 2D display, and may be displayed not only on a normal display device but also on a head-mounted display. Guide display Dgd13 shows a doctor's avatar viewed from behind, allowing users to intuitively understand the movements of both hands, and also displays the avatar's left and right hands. By looking at the avatar, the operator performing the simulated operation can intuitively confirm the movements of the model operation.

[0166] The guide screen Dg14 in Figure 16 displays the internal image PI44, which is the first image read from the operation data recording unit 40, on the right side of the screen, and the guide display Dgd14 on the left side of the screen. The guide display generation unit 63 converts the amount of movement of the physician performing the model operation into the amount of movement of the simulated operation unit 51 based on the model operation data, and displays the simulated operation unit 51 as an animation so that it moves in the same way as the physician performing the model operation. The guide display Dgd14 has the advantage of making the hand movements of the physician performing the model operation easy to understand. For example, by displaying the animation of the model operator's hand near the position of the simulated operator's hand using smart glasses, the model operation can be understood more intuitively.

[0167] The guide screen Dg15 in Figure 17 displays the internal image PI45, which is the first image read from the operation data recording unit 40, on the right side of the screen, and the guide display Dgd15 on the left side of the screen. The guide display generation unit 63 displays the movements of the physician performing the model operation as a video in accordance with the simulated operation as the guide display Dgd15. In the guide display Dgd15, the movement of the right elbow, such as "press slowly," and the movement of the hand, such as "twist to the right," are supplemented by text. The guide display generation unit 63 compares the model operation data and the simulated operation data and adjusts the playback speed of the external image to display the movements of the physician performing the model operation in accordance with the simulated operation. The operator performing the simulated operation can practice insertion by watching the video and imitating the movements of the model operation.

[0168] The guide screen Dg16 in Figure 17 displays the internal image PI46, which is the first image read from the operation data recording unit 40, on the right side of the screen, and the guide display Dgd16 on the left side of the screen. The dashed circle in the internal image PI46 indicates the direction in which the endoscope tip should be pointed. The guide display generation unit 63 displays the direction in which the endoscope tip should be pointed as the guide display Dgd16, along with text indicating the direction in which the endoscope tip should be pointed. The operator performing the simulated operation can receive assistance from the dashed circle in the internal image PI46 and the text in the guide display Dgd16.

[0169] The guide screen Dg17 in Figure 17 displays the internal image PI47 read from the typical scene recording unit 41 of the operation data recording unit 40 on the right side of the screen, and the guide display Dgd17 on the left side of the screen. The guide display generation unit 63 displays the recovery pattern for the case (Case A) that is the target of the simulated operation as the guide display Dgd17. As described above, the typical scene recording unit 41 of the operation data recording unit 40 records the recovery time-series images (internal images) and their operation data that recover from insertion difficulties by model operations in typical scenes such as scenes that are difficult to insert.

[0170] Figure 18 is an explanatory diagram illustrating the contents recorded by the typical scene recording unit 41. As shown in Figure 18, the typical scene recording unit 41 records image patterns in typical scenes such as scenes that are difficult to insert, as well as recovery time-series images (internal images) obtained when the typical scene is recovered, and operation data.

[0171] In the guide screen Dg1 of Figure 17, the current image pattern read from the typical scene recording unit 41 is displayed as the internal image PI47 for practicing typical scenes. In the internal image PI47, the back of the lumen is located at the edge of the screen, and if insertion continues as is, there is a risk of losing sight of the back of the lumen. Therefore, the guide display generation unit 63 reads the recovery time-series image from the typical scene recording unit 41 using the guide display Dgd17 and plays it in a loop. In this case, the guide display generation unit 63 compares the model operation data acquired along with the recovery time-series image with, for example, simulated operation data, and if the difference does not become small, it plays the recovery time-series image in a loop to match the simulated operation. This makes it possible to practice insertion repeatedly using typical patterns such as difficult insertion scenes. Loop playback may also be performed in cases where the model operation is clearly difficult, even without comparing the model operation data and the simulated operation data, and loop playback may also be performed as needed based on the operator's profile and skill level. Note that these guide presentation conditions are not limited to loop playback type guides.

[0172] Furthermore, in S21 of Figure 13, such loop playback may be implemented. This loop playback repeatedly presents guide information to make it easier to confirm the time changes of the guide that represents the operation, for example, loop playback of the movement of an avatar. With loop playback, by repeatedly viewing specific actions or procedures, techniques and knowledge can be understood efficiently. Loop playback may be a loop of continuous images or a loop of video. The selectable guide modes include a loop playback mode that repeatedly presents the operation generated from referring to operation data performed following endoscopic images at specific timings and insertions at specific sites in specific cases.

[0173] Thus, in this embodiment, various guide displays are available, allowing even doctors who are not proficient in the operation to effectively practice insertion while looking at and listening to the guide displays.

[0174] The present invention is not limited to the embodiments described above, and the components can be modified and implemented in practice without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining the multiple components disclosed in the embodiments described above. For example, some components of all the components shown in the embodiments may be deleted. Moreover, components from different embodiments may be appropriately combined.

[0175] Furthermore, many of the controls and functions described here, primarily those explained in flowcharts, can be configured by program, and these controls and functions can be realized by a computer reading and executing the program. The program can be recorded or stored in whole or in part as a computer program product on portable media such as flexible disks, CD-ROMs, and non-volatile memory, or on storage media such as hard disks and volatile memory, and can be distributed or provided at the time of product shipment or via portable media or communication lines. Users can easily implement the endoscopic operation data-based guiding method, guiding device, endoscopic insertion practice method, insertion practice system, insertion practice method, and insertion practice program of this embodiment by downloading and installing the program on a computer via a communication network or installing it on a computer from a recording medium.

Claims

1. A guiding method based on endoscopic operation data, comprising the steps of: acquiring time-dependent change information between operation information of an operator who has previously operated an endoscope for a specific case and an endoscopic image obtained by an imaging unit provided at the tip of the endoscope, and recording the operation information, endoscopic image, and change information in association as endoscopic operation data, the steps being: recording the endoscopic image of the specific case at a first timing and operation information relating to operations performed from the first timing; referring to the endoscopic operation data recorded in the recording step with respect to operations performed from the first timing; and generating an operation guide information based on the referenced endoscopic operation data.

2. The endoscopic operation data-based guiding method according to claim 1, further comprising a selection step of selecting the above-mentioned specific case, wherein the selection step involves selecting profile information of the case subject and the case operator, or making a selection based on the profile information of the operator using this operation guide.

3. The first timing described above is determined by specifying the location of the target organ in the specific case, and the guiding method based on endoscopic operation data according to claim 1.

4. The endoscopic operation data-based guiding method according to claim 1, further comprising a guide mode selection step that allows the above operation guide information to be selected from a plurality of guide modes.

5. The endoscopic operation data-based guiding method according to claim 4, wherein the above selectable guide mode selection step comprises a loop playback mode that repeatedly presents the operation based on the endoscopic operation data relating to the operation performed from the first timing, as a selectable guide mode.

6. A guide device that, for a specific case, acquires time-dependent change information between the operator's operation information of the endoscope that has operated the endoscope in advance and the endoscopic image obtained by an imaging unit provided at the tip of the endoscope, and records the operation information, endoscopic image, and change information in association as endoscopic operation data, comprising: an operation information recording unit that records the endoscopic image of the specific case at a first timing and operation information related to operations performed from the first timing; and a guide generation unit that refers to the endoscopic operation data recorded by the operation information recording unit with respect to operations performed from the first timing and generates operation guide information based on the referenced endoscopic operation data.

7. A guiding method based on endoscopic operation data, comprising the steps of: acquiring time-dependent change information between operation information of an operator who has previously operated an endoscope for a specific case and an endoscopic image obtained by an imaging unit provided at the tip of the endoscope, and recording the operation information, endoscopic image, and change information in association as endoscopic operation data, wherein the steps include: recording the endoscopic image of the specific case at first and second timings and operation information relating to operations performed from the first timing to the second timing; referencing the endoscopic operation data when the endoscopic image of the specific case at the first timing recorded in the recording unit changes to the endoscopic image at the second timing for the same case; and guiding the generation of multiple guide information based on the referenced endoscopic operation data.

8. An endoscope insertion practice method comprising the steps of: acquiring time-dependent change information between operation information of an operator who has previously operated the endoscope and an endoscope image obtained by an imaging unit provided at the tip of the endoscope for a specific case, and recording the operation information, endoscope image, and change information in association as endoscope operation data, the steps being: recording the endoscope image of the specific case at a first timing and operation information relating to operations performed from the first timing; referring to the endoscope operation data recorded in the recording step with respect to operations performed from the first timing; and generating an operation guide method that presents an endoscope operation guide method indicating the operation content for a predetermined unit of operation based on the referenced endoscope operation data.

9. An insertion practice system comprising: a first data acquisition device that acquires and records model operation data obtained by quantifying the amount of operation performed by an endoscope operator; a simulated operation device for simulating the insertion operation of the endoscope insertion section; a second data acquisition device that acquires simulated operation data obtained by quantifying the amount of operation performed by the simulated operation device; a guide generation unit that reads the model operation data and generates guide information indicating the operation content for predetermined operation units; and a guide presentation unit that provides presentations based on the guide information when the simulated operation device is operated.

10. The insertion practice system according to claim 9, further comprising a selection unit for identifying the operation of the endoscope operator before the operation of the above-mentioned simulated operating device and selecting the above-mentioned model operation data corresponding to the identified operation.

11. The insertion practice system according to claim 10, wherein the selection unit selects the model operation data by specifying at least one of the following: the area to be operated on, the age and gender of the subject, and information identifying the endoscope operator who performed the model operation.

12. The insertion practice system according to claim 9, wherein the first data acquisition device acquires and records the model operation data and the endoscopic image obtained when acquiring the model operation data, and the guide generation unit generates guide information for displaying the model operation data and the endoscopic image together.

13. The operation practice system according to claim 9, wherein the first data acquisition device acquires the model operation data based on external images acquired by an external camera that images the endoscope operator performing an insertion or removal operation of an endoscope having an imaging unit at its tip into the body of a subject, and internal images of the body obtained by the imaging unit.

14. The operation practice system according to claim 13, wherein the guide generation unit generates guide information for displaying at least one of the model operation data, the external image, and the internal image.

15. The insertion practice system according to claim 9, wherein the guide generation unit sets the unit of repeated practice of operation on the simulated operating device to the predetermined operation unit.

16. The insertion practice system according to claim 9, wherein the guide generation unit analyzes the operations that caused the image changes between frames, and defines a predetermined unit of operation as a single section in which the operations that caused the image changes were few in number.

17. The insertion practice system according to claim 9, wherein the guide generation unit generates guide information for displaying the model operation data and simulated operation data as graphs, displaying the operation content based on the model operation data as text, displaying the movement of the simulated operator based on the model operation data, displaying the movement of the endoscope operator using an avatar, displaying the movement of the endoscope operator converted into the movement of the simulated operator, displaying the external image, or displaying the direction of operation of the simulated operator.

18. The insertion practice system according to claim 9, wherein the guide generation unit generates guide information for displaying a typical pattern in the insertion operation of the endoscope insertion unit and a time-series pattern that corrects that typical pattern.

19. An insertion practice method for an insertion practice system having a first data acquisition device, a simulated operator, a second data acquisition device, a guide generation unit, and a guide presentation unit, wherein the first data acquisition device acquires and records model operation data obtained by quantifying the amount of operation performed by an endoscope operator; the simulated operator simulates the insertion operation of the endoscope insertion unit; the second data acquisition device acquires simulated operation data obtained by quantifying the amount of operation performed by the simulated operator; the guide generation unit reads the model operation data and generates guide information indicating the operation content for predetermined operation units; and the guide presentation unit provides a presentation based on the guide information when the simulated operator is operated.

20. An insertion practice program for executing a procedure, which involves a computer acquiring and recording model operation data obtained by quantifying the amount of operation performed by an endoscope operator, acquiring simulated operation data obtained by quantifying the amount of operation of the simulated operating device when the insertion operation of the endoscope insertion section is simulated, reading the model operation data and generating guide information indicating the operation content for predetermined operation units, and providing presentations based on the guide information when the simulated operating device is operated.

Images