Program, information processing method, information processing device, and trained model generation method

Abstract

This program causes a computer to execute processing for: acquiring a fluoroscopic image of a patient to whom a treatment related to an artificial hip joint including a cup is to be applied; and inputting the acquired fluoroscopic image to a trained model that has been trained to output, when a fluoroscopic image is inputted thereto, positional information on the anterior wall of the acetabulum and the posterior wall of the acetabulum of the patient, thereby deriving the positional information on the anterior wall of the acetabulum and the posterior wall of the acetabulum of the patient.

Description

Program, Information Processing Method, Information Processing Apparatus, and Method for Generating Learning Model

[0001] The present invention relates to a program, an information processing method, an information processing apparatus, and a method for generating a learning model. This application claims priority based on Japanese Application No. 2024-195332 filed on November 7, 2024, and incorporates all the descriptions described in the Japanese application.

[0002] An acetabular socket angle setting device in hip arthroplasty is known (for example, Patent Document 1). According to the acetabular socket angle setting device in hip arthroplasty described in Patent Document 1, an acetabular socket angle setting method based on the pelvic plane and the inter-teardrop line is provided.

[0003] Japanese Patent Application Laid-Open No. 2022-031047

[0004] However, in the acetabular socket angle setting device in hip arthroplasty of Document 1, regarding deriving the position information of the anterior wall and the posterior wall of the acetabulum of the patient, which is useful information when shaving the pelvis to place the cup included in the hip joint, for a fluoroscopic image of a patient undergoing a procedure related to the hip joint, it is not considered.

[0005] In view of such circumstances, the present disclosure has been made, and an object thereof is to provide a program or the like that can derive the position information of the anterior wall and the posterior wall of the acetabulum of a patient based on a fluoroscopic image of a patient undergoing a procedure related to the hip joint.

[0006] In one approach, the program acquires fluoroscopic images of a patient undergoing artificial hip joint surgery, and when these images are input to a trained model, the program derives positional information for the inner plate, anterior wall of the acetabulum, and posterior wall of the acetabulum by inputting the acquired fluoroscopic images. Based on the derived positional information for the inner plate, the program recognizes the inner plate line in the patient's pelvis in the acquired fluoroscopic images, indicating the point where the pelvis is shaved down to accommodate the cup included in the artificial hip joint. Based on the positional information of the anterior and posterior walls of the acetabulum obtained, the system recognizes acetabular positional relationship information regarding the positional relationship between the cup and the acetabulum when the cup is placed in the patient's pelvis in the acquired fluoroscopic image. The system derives the cup CE angle or bony coverage rate when the placement of the cup is assumed to be in a state where the patient's pelvis has been shaved using a bone excavation reamer used in the procedure for the artificial hip joint. The system then overlays the recognized inner plate line and the acetabular positional relationship information onto the fluoroscopic image and executes a process to display the derived cup CE angle or bony coverage rate attached to the fluoroscopic image.

[0007] One proposed method involves a computer acquiring fluoroscopic images of a patient undergoing artificial hip joint treatment, and inputting the acquired fluoroscopic images into a learning model trained to output positional information of the inner plate, anterior wall of the acetabulum, and posterior wall of the acetabulum. The learning model then derives positional information of the inner plate, anterior wall of the acetabulum, and posterior wall of the acetabulum from the patient, and based on the derived positional information of the inner plate, it recognizes the inner plate line in the patient's pelvis in the acquired fluoroscopic image, indicating the point of maximum cutting when cutting the pelvis to position the cup included in the artificial hip joint. Based on the derived positional information of the anterior and posterior walls of the acetabulum, the system recognizes acetabular positional relationship information regarding the positional relationship between the cup and the acetabulum when the cup is placed in the patient's pelvis in the acquired fluoroscopic image. The system then derives the cup CE angle or bony coverage rate when the patient's pelvis is shaved down by a bone-excavating reamer used in the procedure for the artificial hip joint, and overlays the recognized inner plate line and acetabular positional relationship information onto the fluoroscopic image. The system then performs a process to display the derived cup CE angle or bony coverage rate attached to the fluoroscopic image.

[0008] One proposed design includes an information processing device comprising: an acquisition unit that acquires fluoroscopic images of a patient undergoing artificial hip joint treatment; a position information derivation unit that derives position information of the inner plate, anterior wall of the acetabulum, and posterior wall of the acetabulum of the patient by inputting the acquired fluoroscopic images into a learning model that has been trained to output position information of the inner plate, anterior wall of the acetabulum, and posterior wall of the acetabulum when an fluoroscopic image is input; and an inner plate line recognition unit that recognizes an inner plate line indicating the point of maximum cutting when cutting the pelvis to position the cup included in the artificial hip joint, based on the derived position information of the inner plate, in the patient's pelvis in the acquired fluoroscopic image. Based on the positional information of the anterior and posterior walls of the acetabulum obtained, the device includes: an acetabular positional relationship information recognition unit that recognizes acetabular positional relationship information relating to the positional relationship between the cup and the acetabulum when the cup is placed in the patient's pelvis in the acquired fluoroscopic image; a derivation unit that derives the cup CE angle or bony coverage rate when the placement of the cup is assumed to be in a state where the patient's pelvis has been shaved by a bone excavation reamer used in the procedure for the artificial hip joint; and a display unit that superimposes the recognized inner plate line and the acetabular positional relationship information onto the fluoroscopic image and displays the derived cup CE angle or bony coverage rate attached to the fluoroscopic image.

[0009] One proposed method for generating a learning model involves acquiring training data including fluoroscopic images of a patient undergoing artificial hip joint surgery and positional information of the patient's inner plate. Based on the acquired training data, a learning model is generated that outputs positional information of the inner plate when a fluoroscopic image is input. The fluoroscopic image includes multiple pseudo-fluoroscopic images generated at multiple different fluoroscopic angles from a DRR image converted from the patient's tomographic image.

[0010] According to this disclosure, it is possible to provide a program that derives positional information of the patient's inner plate, anterior wall of the acetabulum, and posterior wall of the acetabulum based on fluoroscopic images of the patient undergoing artificial hip joint surgery.

[0011] This is a schematic diagram showing an overview of an intraoperative support system including an information processing device according to Embodiment 1. This is a block diagram showing an example of the configuration of the information processing device. This is an explanatory diagram showing an example of a learning model. This is a flowchart showing an example of the processing procedure of the processing unit of the information processing device (during model learning). This is an explanatory diagram showing an example of a second learning model (bony coverage model). This is a flowchart showing an example of the processing procedure of the processing unit of the information processing device (during model operation). This is an explanatory diagram regarding the cup CE angle. This is an explanatory diagram regarding the inner plate line (limit depth). This is an explanatory diagram regarding acetabular positional relationship information (anterior and posterior wall region information, central region information). This is an explanatory diagram regarding the mandatory line. This is an explanatory diagram regarding the upward limit line. This is an explanatory diagram regarding the reachable area (strike zone). This is an explanatory diagram regarding the anterior tilt angle and abduction angle of the cup (assumed placement when the pelvis has been shaved by a bone excavation reamer). This is an explanatory diagram illustrating the display screen of support information (support information superimposed screen). This is an explanatory diagram illustrating the display screen of support information (distortion correction screen).

[0012] The present invention will be described in detail below with reference to the drawings illustrating its embodiments. (Embodiment 1)

[0013] Figure 1 is a schematic diagram showing an overview of an intraoperative support system S including an information processing device 1 according to Embodiment 1. Figure 2 is a block diagram showing an example of the configuration of the information processing device 1. The intraoperative support system S is configured with the information processing device 1 as the main device, and the information processing device 1 is communicably connected to a fluoroscopic image acquisition device such as an X-ray device 62, and a tomographic image acquisition device such as a CT device 61 when generating a learning model 101 (position information model) or a second learning model 102 (bone coverage model). When the information processing device 1 outputs information related to intraoperative support using the learning model 101 or the second learning model 102, it acquires X-ray images (fluoroscopic images) taken by the X-ray device 62 in real time. When the information processing device 1 generates the learning model 101 or the second learning model 102, it acquires CT images (tomographic images) taken by the CT device 61. The information processing device 1 may also be communicably connected to an electronic medical record server that stores and manages various medical data related to patient K.

[0014] Patient K, who is undergoing treatment for an artificial hip joint 8, has X-ray images (fluoroscopic images) taken by an X-ray device 62 while the pelvis is being excavated with a bone excavation reamer 7 (during surgery). Therefore, the X-ray images will include the bone excavation reamer 7 excavating the pelvis (during excavation). The X-ray device 62 is configured to be rotatable relative to patient K, and may move circumferentially over, for example, 180 degrees or 360 degrees with the patient K's height direction (height direction) as the axis of rotation (Z axis). By rotating the X-ray device 62 with respect to the patient K's body axis (Z axis) in this way and taking X-ray images (fluoroscopic images) of patient K, multiple X-ray images (fluoroscopic images) can be obtained for patient K at different fluoroscopic angles (X-ray irradiation angles). Therefore, the radiographic images (fluoroscopic images) include a radiographic image of patient K's pelvis taken from the front and radiographic images of patient K's pelvis taken from the left and right sides.

[0015] After the pelvis is properly excavated by a physician using a bone excavation reamer 7, the artificial hip joint 8, including the cup 81, is inserted (implanted) into the patient K's body. The artificial hip joint 8 includes a hemispherical cup 81 that fits into the excavated pelvis and a stem that is inserted into the femur. The bone excavation reamer 7 has a hemispherical tip for excavation, and the outer edge shape of this tip is the same shape and size as the outer edge shape of the cup 81 of the artificial hip joint 8.

[0016] As will be described in detail later, the processing unit 2 of the information processing device 1 recognizes the shape of the bone drilling reamer 7 (the hemispherical tip that performs drilling) included in the fluoroscopic image (X-ray image) captured and acquired during surgery, using, for example, an object detection model such as YOLO or edge detection, and identifies the position and inclination of the bone drilling reamer 7 in the pelvis. The processing unit 2 of the information processing device 1 considers the identified position and inclination of the bone drilling reamer 7 as the position and inclination of the cup 81 that is expected to be positioned when the pelvis is drilled by the bone drilling reamer 7, and calculates and outputs various information regarding the cup 81 that is expected to be positioned. Furthermore, the processing unit 2 of the information processing device 1 uses the inner plate line derived by the learning model 101 based on the fluoroscopic image (X-ray image) captured and acquired during or immediately before surgery to superimpose and display the inner plate line (reachable area) to support drilling by the bone drilling reamer 7 on the fluoroscopic image (X-ray image) captured and acquired during surgery. Furthermore, the processing unit 2 of the information processing device 1 may use acetabular positional relationship information derived by the learning model 101 based on fluoroscopic images (X-ray images) captured and acquired during or immediately before surgery to superimpose and display acetabular positional relationship information on the fluoroscopic images (X-ray images) captured and acquired during surgery to support excavation by the bone excavation reamer 7. In this case, the acetabular positional relationship information may include anterior-posterior wall region information indicating the areas of the anterior wall (acetabular anterior wall) and posterior wall (acetabular posterior wall) of the acetabulum that the front and back of the cup 81 contact (or can contact) when the cup 81 is placed, and this anterior-posterior wall region information makes it possible to recognize the contact relationship between the anterior wall and posterior wall of the acetabulum and the cup 81 in the pelvis of patient K, so that the cup 81 can obtain sufficient initial fixation. Alternatively, the acetabular positional relationship information may include central region information indicating the region where the center of the cup 81 is located when the cup 81 is placed in contact with the anterior and posterior walls of the acetabulum. Similar to the anterior and posterior wall region information, this allows for the recognition of the contact relationship between the anterior and posterior walls of the acetabulum and the cup 81, which is necessary for the cup 81 to achieve sufficient initial fixation.

[0017] The information processing device 1 is a computer capable of various information processing and information transmission and reception, such as a server device or a personal computer. The server device includes not only a single server device but also a cloud server device or virtual server device composed of multiple computers. If the information processing device 1 is configured as a cloud server device, for example, the information processing device 1 does not need to be installed in the medical facility where the patient K is located, similar to medical equipment such as a CT scanner 61 or an X-ray machine 62, and may be connected to these medical devices via an external network such as the Internet. The information processing device 1 includes a processing unit 2, a storage unit 3, an input / output interface 4, and a communication unit 5.

[0018] The processing unit 2 has one or more arithmetic processing units equipped with timing functions such as CPUs (Central Processing Units), MPUs (Micro-Processing Units), and GPUs (Graphics Processing Units), and performs various information processing, control processing, etc. related to the information processing unit 1 by reading and executing programs P (program products) stored in the storage unit 3.

[0019] The storage unit 3 includes volatile storage areas such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), and flash memory, as well as non-volatile storage areas such as EEPROM or hard disk. The storage unit 3 pre-stores programs P (program products) and data referenced during processing. The programs P (program products) stored in the storage unit 3 may be programs P (program products) read from a recording medium M that the information processing device 1 can read. Alternatively, programs P (program products) may be downloaded from an external computer (not shown) connected to a communication network (not shown) and stored in the storage unit 3.

[0020] The memory unit 3 stores various medical data related to patient K and various values ​​determined in the preoperative plan. The memory unit 3 also stores the actual files that constitute the learning model 101 and the second learning model 102. These actual files may be configured as part of the program P (program product).

[0021] The communication unit 5 is a communication module or communication interface for communicating with an electronic medical record server or an information terminal such as a smartphone held by a medical professional, either by wire or wireless connection. For example, it may be a wired communication module such as an Ethernet® connector, a narrow-area wireless communication module such as Wi-Fi® or Bluetooth®, or a wide-area wireless communication module such as 4G or 5G. The processing unit 2 communicates with the electronic medical record server or information terminal via the communication unit 5, for example, via a local network within the medical institution or an external network such as the Internet.

[0022] The input / output interface 4 is a communication interface compliant with a communication standard such as RS232C or USB. An input device such as a keyboard or a display device 41 such as a liquid crystal display is connected to the input / output interface 4. Furthermore, medical equipment such as a CT scanner 61 or an X-ray scanner 62 may also be connected to the input / output interface 4.

[0023] Figure 3 is an explanatory diagram showing an example of a learning model 101. The learning model 101 is a neural network (NN) that performs object detection, semantic segmentation, or instance segmentation, and is composed of, for example, RCNN (Regions with Convolutional Neural Network), Fast RCNN, Faster RCNN, or SSD (Single Shot Multibook Detector), YOLO (You Only Look Once), etc.

[0024] The learning model 101 outputs positional information of the inner plate and acetabulum in the pelvis, based on the input fluoroscopic image (X-ray image). The positional information of the inner plate may be indicated, for example, by a linear inner plate line indicating the periphery of the inner plate in the pelvis, or by a plurality of points constituting the inner plate line (a point cloud consisting of a plurality of inner plate corresponding points). The inner plate line is a line (boundary line) that indicates the limit depth when excavating the pelvis, and the learning model 101 functions as a limit depth extraction model that extracts the limit depth included in the input fluoroscopic image (X-ray image). The positional information of the acetabulum may be indicated, for example, by the regions of the anterior wall (anterior acetabular wall) and posterior wall (posterior acetabular wall) located anteriorly and posteriorly in the pelvis. Furthermore, the positional information of the acetabulum may be indicated by the region where the center of the cup 81 is located when the cup 81 is placed in contact with the anterior and posterior walls of the acetabulum. Furthermore, the positional information of the acetabulum may include the region of the load-bearing surface located between the anterior and posterior walls. Thus, the learning model 101 functions as a position information model that outputs position information of the medial plate and acetabulum in the pelvis. As described above, a fluoroscopic image (X-ray image) includes multiple fluoroscopic images with different fluoroscopic angles, and the learning model 101 may output position information of the medial plate (medial plate line), position information of the acetabulum (anterior and posterior wall regions, central region of cup 81), or position information of both (position information of the medial plate and acetabulum) depending on the fluoroscopic angle of the input fluoroscopic image. For example, if the fluoroscopic angle is set to image from the front of the pelvis, the learning model 101 may output position information of the medial plate (medial plate line), and if the fluoroscopic angle is set to image from the side of the pelvis, it may output position information of the acetabulum (anterior and posterior wall regions, central region of cup 81).

[0025] If the learning model 101 (location information model) is composed of a neural network including a CNN (Convolutional Neural Network) that extracts image features, such as an RCNN, the input layer included in the learning model 101 has multiple neurons that accept input of image pixel values ​​and passes the input pixel values ​​to the hidden layer. The hidden layer has multiple neurons that extract image features and passes the extracted image features to the output layer. The output layer has one or more neurons that output region information including the position of the inner platen line (point cloud corresponding to the inner platen), and outputs the position of the inner platen line (point cloud corresponding to the inner platen) (region coordinates or pixel number, etc.) based on the image features output from the hidden layer. Furthermore, the output layer has one or more neurons that output region information including the position of the anterior wall and posterior wall of the acetabulum, and outputs the position of the anterior wall and posterior wall of the acetabulum (region coordinates or pixel number, etc.) based on the image features output from the hidden layer. Alternatively, the output layer may have one or more neurons that output region information, including the position of the center of the cup 81, when the cup 81 is positioned in contact with the anterior and posterior walls of the acetabulum, and output the position of the center of the cup 81 (region coordinates or pixel number, etc.) based on the image features output from the intermediate layer. The center of the cup 81 is not limited to the center of the cup 81 itself, but may include a region that defines the center of the cup 81.

[0026] The neural network (learning model 101) trained using training data is intended to be used as a program module, which is part of artificial intelligence software. The learning model 101 is used in an information processing device 1 that includes a processing unit 2 (CPU, etc.) and a memory unit 3, as described above. By being executed in the information processing device 1, which has such computational processing capabilities, a neural network system is formed. Specifically, the processing unit 2 of the information processing device 1 performs calculations to extract feature quantities from the image input to the input layer according to commands from the learning model 101 stored in the memory unit 3, and outputs the position (region) of the inner plate line (inner plate corresponding point cloud) from the output layer.

[0027] The learning model 101 can be generated by preparing training data in which problem data consisting of fluoroscopic images (X-ray images) including the pelvis (hip joint) are associated with labels (answer data) that indicate the position (region) of the medial plate line in the pelvis or the anterior and posterior walls of the acetabulum, and then using this training data to machine-learn an untrained neural network. In this case, the fluoroscopic images (X-ray images) may be pseudo-fluoroscopic images (DRR-derived fluoroscopic images [DRR-F]) generated using DRR images (Digital Reconstructed Radiograph: CT reconstruction simulation images) converted from tomographic images such as CT images including the pelvis (hip joint).

[0028] Training data is stored, for example, in the memory unit 3 of the information processing device 1 and can be generated by aggregating images and physician's findings contained in a large amount of diagnostic or surgical results stored on an electronic medical record server in a medical institution such as a hospital. In other words, the medial plate line in the pelvis (medial plate line, or the point group corresponding to the medial plate line that constitutes the medial plate line) and the anterior and posterior walls of the acetabulum are locations (internal body parts) identified based on findings by physicians, etc. According to the learning model 101 that has been learned and configured in this way, by inputting a fluoroscopic image (X-ray image) into the learning model 101, it is possible to obtain information indicating the position of the medial plate line of the pelvis (region coordinates in the image coordinate system) and the anterior and posterior walls of the acetabulum contained in the fluoroscopic image. The learning model 101 may also output a fluoroscopic image in which the derived medial plate line and the anterior and posterior walls of the acetabulum are superimposed on the input fluoroscopic image (X-ray image).

[0029] In this embodiment, the learning model 101 is described as being an RCNN, but the learning model 101 is not limited to an RCNN, and may be a learning model 101 constructed with other learning algorithms such as neural networks other than RCNNs, SVMs (Support Vector Machines), transformers, YOLO, Bayesian networks, regression trees, etc. The dataset of problem data and answer data included in the training data for learning the learning model 101 and the dataset of input data and output data when using the learning model 101 are synonymous, and if defined in one dataset, it will naturally apply to the other dataset as well.

[0030] Figure 4 is a flowchart showing an example of the processing procedure (during model learning) of the processing unit 2 of the information processing device 1. The processing unit 2 of the information processing device 1 receives operator input, for example, via a keyboard connected to the input / output I / F 4, and performs the following processing based on the received operation.

[0031] The processing unit 2 of the information processing device 1 acquires tomographic images (S11). In generating the learning model 101, the processing unit 2 of the information processing device 1 acquires tomographic images (CT images) of the pelvises of a large number of subjects. These tomographic images (CT images) are taken on a subject or patient K using a CT scanner 61 and consist of multiple images (group of tomographic images) showing cross-sections of the body with a predetermined slice width. These tomographic images (CT images) are stored in an electronic medical record server or the like at a medical institution such as a hospital. The processing unit 2 of the information processing device 1 can acquire a large number of tomographic images (CT images) by accessing the electronic medical record server or the like.

[0032] The processing unit 2 of the information processing device 1 converts the acquired tomographic image into a DRR image (S12). The processing unit 2 of the information processing device 1 converts the tomographic image into a DRR image (Digital Reconstructed Radiograph: CT reconstruction simulation image), associates the converted DRR image with the original tomographic image data (a group of tomographic images showing cross-sections of the body at a predetermined slice width), and stores it in the storage unit 3. The processing unit 2 of the information processing device 1 may, for example, convert the tomographic image into a DRR image using ray casting processing in which the line of sight is replaced with an X-ray source.

[0033] The processing unit 2 of the information processing device 1 generates a pseudo-fluoroscopic image using the DRR image (S13). When generating a pseudo-fluoroscopic image using the DRR image, the processing unit 2 of the information processing device 1 may generate multiple pseudo-fluoroscopic images (DRR-F) with different fluoroscopic angles, i.e., different X-ray irradiation angles for the subject. This makes it possible to generate multiple pseudo-fluoroscopic images with different fluoroscopic angles from a single DRR image, and to efficiently generate multiple times the number of pseudo-fluoroscopic images compared to the original tomographic image (CT image), i.e., to perform training data augmentation (data argumentation).

[0034] The processing unit 2 of the information processing device 1 may perform angle correction on the original data, which is a tomographic image (CT image) or DRR image, when generating a pseudo-fluoroscopic image. The angle correction is performed so that the pseudo-fluoroscopic image (DRR-F) shows a complete front view of the pelvis, and the processing unit 2 of the information processing device 1 may modify the tomographic image (CT image) or DRR image based on the correction value of the angle correction.

[0035] The processing unit 2 of the information processing device 1 generates training data using a pseudo-transparency image (S14). The processing unit 2 of the information processing device 1 generates training data by adding (annotating) positional information of the pelvic inner plate and acetabulum contained in the pseudo-transparency image to the pseudo-transparency image. The positional information of the pelvic inner plate may be indicated, for example, by a linear inner plate line indicating the periphery of the pelvic inner plate, or by a plurality of points (inner plate corresponding points) that constitute the inner plate line. The positional information of the pelvic acetabulum may be indicated, for example, by an elliptical object indicating the anterior and posterior wall regions of the pelvis. Alternatively, the positional information of the pelvic acetabulum may be indicated, for example, by a circular object indicating the center of the cup 81 located inside the acetabulum of the pelvis. The positional information of the inner plate (inner plate line) and the positional information of the acetabulum (anterior-posterior wall region information, central region information) assigned to the pseudo-fluoroscopic image are determined in accordance with the locations (internal body parts) identified by physicians or other medical professionals based on their findings in the tomographic image (CT image), which is the source data for the pseudo-fluoroscopic image.

[0036] The processing unit 2 of the information processing device 1 may, when generating training data, annotate the positional information of the inner plate and acetabulum in the pelvis to a pseudo-fluoroscopic image based on tomographic images (CT images) on which the inner plate correspondence points (landmarks) and the regions of the anterior and posterior walls of the acetabulum have been marked by a physician or the like. In this case, the processing unit 2 of the information processing device 1 can align the DRR image converted from the tomographic image (CT image) and the pseudo-fluoroscopic image (DRR-F) generated from the DRR image by applying the same internal coordinate system. Therefore, the processing unit 2 of the information processing device 1 may, based on the internal coordinate system, expand the inner plate correspondence points (landmarks) and the regions of the anterior and posterior walls, which have been marked on each of a plurality of tomographic images with a predetermined slice width, onto the pseudo-fluoroscopic image, thereby annotating the positional information of the inner plate (inner plate line) and the positional information of the acetabulum in the pseudo-fluoroscopic image.

[0037] The processing unit 2 of the information processing device 1 generates a learning model 101 using the acquired training data (S15). The processing unit 2 of the information processing device 1 generates a learning model 101 by training an untrained neural network by applying the training data (images in which the positional information of the inner plate and acetabulum is annotated to pseudo-transspective images).

[0038] In this embodiment, the learning model 101 outputs positional information of the inner plate of the pelvis (inner plate line) and positional information of the acetabulum included in the input pseudo-fluoroscopic image, but is not limited to this. The learning model 101 may also output a cup position derived based on the positional information of the inner plate of the pelvis (inner plate line) and positional information of the acetabulum included in the input pseudo-fluoroscopic image. In this case, the learning model 101 may also output positional information (outer edge diagram of the cup 81 at the installation position when the cup 81 is installed) indicating a cup position that has a required line and cup CE angle, or bony coverage rate, based on the input pseudo-fluoroscopic image. In other words, the learning model 101 may output positional information indicating the cup position (a diagram of the outer edge of the cup 81 at the installation position when the cup 81 is installed) by receiving a pseudo-fluoroscopic image, a required line or bone coverage rate calculated based on the pseudo-fluoroscopic image, and the radius of the cup 81 and the target CE angle determined in the preoperative plan. In this case, the learning model 101 may output a fluoroscopic image in which the derived cup position (diagram of the outer edge of the cup 81) is superimposed on the input fluoroscopic image (X-ray image). In this case, the learning model 101 is trained to output the cup position based on these input data (pseudo-fluoroscopic image, required line, bone coverage rate, radius of the cup 81, target CE angle).

[0039] Figure 5 is an explanatory diagram showing an example of the second learning model 102 (bone coverage model). The second learning model 102 is composed of, for example, a CNN (Convolutional Neural Network). Based on the input fluoroscopic image (X-ray image), the second learning model 102 outputs the bone coverage ratio of the pelvis contained in the fluoroscopic image. The bone coverage ratio includes, for example, the BCI (Bone Coverage Index) or the three-dimensional bone coverage ratio. The BCI is defined, for example, as the ratio of the horizontal width of the covered portion of the cup 81 to the horizontal width of the entire cup 81. The three-dimensional bone coverage ratio is a ratio that shows in three dimensions how much of the femoral head is covered by the acetabulum (femoral head coverage state by the acetabulum). In this way, the second learning model 102 functions as a bone coverage model that outputs the bone coverage ratio (BCI, three-dimensional bone coverage ratio) of the pelvis.

[0040] The second learning model 102, like the learning model 101, is trained on multiple perspective images with different perspective angles and consists of a neural network including a CNN (Convolutional Neural Network) that extracts image features. The input layer in the second learning model 102 has multiple neurons that accept image pixel values ​​as input and passes the input pixel values ​​to the hidden layer. The hidden layer has multiple neurons that extract image features from the image and passes the extracted image features to the output layer. The output layer has one or more neurons that output bone coverage and outputs bone coverage based on the image features output from the hidden layer.

[0041] The second learning model 102 can be generated by preparing training data in which problem data consisting of fluoroscopic images (X-ray images) including the pelvis (hip joint) are associated with labels (answer data) that indicate the bone coverage rate in the pelvis, and then using this training data to machine-learn an untrained neural network. The annotation of the bone coverage rate is performed based on the findings of a physician or other medical professional, as in the learning model 101. In this case, as in the learning model 101, the fluoroscopic images (X-ray images) may also be pseudo-fluoroscopic images (DRR-derived fluoroscopic images [DRR-F]) generated using DRR images (Digital Reconstructed Radiograph: CT reconstruction simulation images) converted from tomographic images such as CT images including the pelvis (hip joint).

[0042] Figure 6 is a flowchart showing an example of the processing procedure (during model operation) of the processing unit 2 of the information processing device 1. The processing unit 2 of the information processing device 1 starts the processing shown in the flowchart, for example, when a perspective image is input as a trigger, or when it receives a start command from an input device such as a keyboard connected to the input / output I / F 4.

[0043] The processing unit 2 of the information processing device 1 obtains various values ​​determined in the preoperative plan by referring to the storage unit 3 (S101). The various values ​​determined in the preoperative plan include, for example, the radius of the cup 81 of the artificial hip joint 8 to be placed in the body, the target CE angle, the target bony coverage rate, the target distance between the outer edge and the inner plate line of the cup 81 which has been set in advance as a target value, the target anterior tilt angle, and the target abduction angle.

[0044] The target CE angle indicates the cup CE angle when placing the cup 81 preset as a target value. The target distance indicates the distance between the cup 81 preset as a target value and the inner plate line derived using the learning model 101 based on the tomographic image. The target anteversion angle indicates the anteversion angle of the cup 81 when placing the cup 81 preset as a target value. The target abduction angle indicates the abduction angle of the cup 81 when placing the cup 81 preset as a target value. These various values determined in the preoperative plan are determined by a doctor or the like, pre-input into the information processing apparatus 1, and stored in the storage unit 3. Alternatively, the processing unit 2 of the information processing apparatus 1 may acquire various values determined in the preoperative plan for the patient K based on the patient ID or the like from the electronic medical record system.

[0045] During the operation of the patient K on whom the treatment regarding the artificial hip joint 8 is performed, the processing unit 2 of the information processing apparatus 1 acquires a fluoroscopic image (X-ray image) of the pelvis of the patient K (S102). During the operation of the patient K on whom the treatment regarding the artificial hip joint 8 is performed, that is, in a state where the pelvis of the patient K is being drilled by the bone drilling reamer 7, the processing unit 2 of the information processing apparatus 1 acquires in real time a fluoroscopic image (X-ray image) of the pelvis of the patient K. When acquiring the fluoroscopic image (X-ray image) in real time, the processing unit 2 of the information processing apparatus 1 may acquire it in a video form.

[0046] The fluoroscopic images (X-ray images) obtained in the video form include a fluoroscopic image (X-ray image) of the pelvis of patient K before being drilled by the bone drilling reamer 7, i.e., without the bone drilling reamer 7, and a fluoroscopic image (X-ray image) including the bone drilling reamer 7. Thus, the fluoroscopic images (X-ray images) that capture patient K in real time during the surgery include the bone drilling reamer 7 for shaving the pelvis, and the current position and inclination of the bone drilling reamer 7 (outer edge of the hemispherical tip for drilling) in the pelvis can be specified. When assuming the arrangement of the cup 81 at the current position of the bone drilling reamer 7 in the pelvis, i.e., in the depression (installation space) of the bone drilled by the bone drilling reamer 7, the cup CE angle, the anteversion angle, and the abduction angle of the assumed arranged cup 81 can be derived.

[0047] Figure 7 is an explanatory diagram regarding the cup CE angle. In the illustration of the present embodiment, a semicircular graphic object showing the cup 81 with an assumed arrangement is superimposed and displayed on the fluoroscopic image based on the current position of the bone drilling reamer 7 in the pelvis. The processing unit 2 of the information processing apparatus 1 derives, as the cup CE angle, the angle formed by the line connecting the intersection of the outer edge of the cup 81 and the white line (acetabular sourcil) of the pelvic load surface from the center (COR: Center of Rotation) of the cup 81 assumed to be arranged in the state where the pelvis has been drilled by the bone drilling reamer 7, and the line connecting the lower ends of the left and right teardrop marks in the pelvis with respect to the perpendicular line (Y-axis) to the pelvic reference line.

[0048] The processing unit 2 of the information processing apparatus 1 derives the position information (acetabulum position relationship information) of the inner plate line and the anterior and posterior walls of the acetabulum in the fluoroscopic image using the learning model 101 (S103). The processing unit 2 of the information processing apparatus 1 may input, for example, a fluoroscopic image (X-ray image) captured and obtained immediately before the surgery, i.e., a fluoroscopic image (X-ray image) of the pelvis of patient K before being drilled by the bone drilling reamer 7, into the learning model 101.

[0049] The learning model 101 outputs a fluoroscopic image in which the position information of the inner plate (inner plate line) and the position information of the acetabulum (acetabular positional relationship information) are superimposed on the input fluoroscopic image (X-ray image). The inner plate line indicates the maximum cutting point when cutting the pelvis, i.e., the drilling limit line (limit depth) that must not be exceeded. The position information of the acetabulum (acetabular positional relationship information) includes anterior-posterior wall region information indicating the regions of the anterior wall (acetabular anterior wall) and posterior wall (acetabular posterior wall) of the acetabulum, or central region information indicating the region where the center of the cup 81 is located when the cup 81 is placed in contact with the anterior and posterior walls of the acetabulum, and this information indicates the contact relationship between the anterior and posterior walls of the acetabulum and the cup 81 in order for the cup 81 to obtain sufficient initial fixation. The processing unit 2 of the information processing device 1 derives (identifies) the inner plate line of the pelvis and the regions of the anterior and posterior walls of the acetabulum in the fluoroscopic image based on the output result (inner plate position information) from the learning model 101.

[0050] Figure 8 is an explanatory diagram relating to the inner plate line (critical depth). In the illustration of this embodiment, multiple (three in this embodiment) tomographic images are shown, and each tomographic image (CT image) has a correspondence relationship (same Y coordinate) with the fluoroscopic image (X-ray image) at multiple locations in the Y-axis direction (vertical direction of the human body). Each of these tomographic images was used when generating the learning model 101, and circles indicating inner plate corresponding points (landmarks) are superimposed on each tomographic image (CT image). That is, the pseudo-fluoroscopic image created using the DRR image converted from the tomographic image is given an inner plate line formed by connecting the inner plate corresponding points (landmarks) shown in these multiple tomographic images, and is used as training data when generating the learning model 101.

[0051] The processing unit 2 of the information processing device 1 derives a required line in the fluoroscopic image according to the radius of the cup 81 and the target CE angle (S104). Alternatively, the processing unit 2 of the information processing device 1 may derive a required line in the fluoroscopic image according to the radius of the cup 81 and a predetermined target bony coverage ratio for stabilizing the positioned cup 81. The radius of the cup 81 of the artificial hip joint 8 to be inserted into patient K and the target CE angle or target bony coverage ratio are stored in the storage unit 3 as various values ​​determined in the preoperative plan. The processing unit 2 of the information processing device 1 derives a required line in the fluoroscopic image based on the white line (Acetabular sourcil) formed by the inner edge of the pelvic load surface included in the fluoroscopic image, according to the radius of the cup 81 and the target CE angle or target bony coverage ratio. The white line (Acetabular sourcil) formed by the inner edge of the pelvic load surface can be identified (its position determined) by performing shape recognition processing on the fluoroscopic image. The required line indicates the minimum cutting point when cutting the pelvis in patient K to position the cup 81.

[0052] Figure 9 is an explanatory diagram relating to acetabular positional relationship information (anterior-posterior wall region information, central region information). In the illustration of this embodiment, annotations for generating the learning model 101 are added to an X-ray image (fluoroscopic image acquisition) taken at a fluoroscopic angle from the side of the pelvis, including the regions of the anterior wall (acetabular anterior wall) and posterior wall (acetabular posterior wall), and these are used as training data when generating the learning model 101. In the X-ray image (fluoroscopic image acquisition) taken from the side of the pelvis, the acetabulum is shown as a circle, and the anterior and posterior walls of the acetabulum are located in front of and behind the acetabulum. When the cup 81 is placed, the front and back of the cup 81 can come into contact with the anterior and posterior walls of the acetabulum, and the contact relationship between the anterior and posterior walls of the acetabulum and the cup 81 is recognized by the regions of the posterior and anterior walls (anterior-posterior wall region information). Furthermore, when the cup 81 is positioned, the region on the inside of the acetabulum where the center of the cup 81 may be located is shown as the central region of the cup 81 (central region information). Between the posterior wall region and the anterior wall region lies the load-bearing surface region. The learning model 101 may output the input X-ray image (fluoroscopic image acquisition) with the posterior and anterior wall regions (anterior-posterior wall region information), or the central region of the cup 81 (central region information) superimposed. Furthermore, the learning model 101 may also output the load-bearing surface region with the input X-ray image (fluoroscopic image acquisition) superimposed.

[0053] Furthermore, the processing unit 2 of the information processing device 1 may, for example, input a fluoroscopic image (X-ray image) captured and acquired immediately before the surgery, that is, a fluoroscopic image (X-ray image) of patient K's pelvis before it was excavated by the bone excavation reamer 7, into the second learning model 102. The second learning model 102 outputs a bone coverage ratio, including BCI or three-dimensional bone coverage ratio, for the input fluoroscopic image (X-ray image). The processing unit 2 of the information processing device 1 may also derive the bone coverage ratio (BCI or three-dimensional bone coverage ratio) by inputting multiple fluoroscopic images (X-ray images) taken at different fluoroscopic angles into the second learning model 102.

[0054] Figure 10 is an explanatory diagram regarding the required line. The storage unit 3 of the information processing device 1 stores various values ​​(parameters) determined in the preoperative plan when performing a procedure on the artificial hip joint 8, such as the minimum CE angle (e.g., 10 degrees) set in advance to stabilize the cup 81 to be placed, and the radius of the cup 81 to be implanted. The processing unit 2 of the information processing device 1 obtains these various values ​​determined in the preoperative plan by referring to the storage unit 3.

[0055] The processing unit 2 of the information processing device 1 defines the center (x,y) of the cup 81, which is assumed to be placed in the bone cavity (installation space) excavated by the bone excavation reamer 7, and the intersection point (X,Y) of the white line (Acetabular sourcil) representing the pelvic load surface at the time of excavation and the edge of the cup. In this case, the center (x,y) of the cup 81 is expressed by the following formula using the intersection point (X,Y), the radius (r) of the cup 81, and the cup CE angle (Θ). The X coordinate (x) of the center of the cup 81 is calculated by subtracting the value obtained by multiplying the radius (r) of the cup 81 by the sine of the cup CE angle (Θ) from the X coordinate (X) of the intersection point (x) (x = X - r*sinΘ). The Y-coordinate (y) of the center of cup 81 is calculated by subtracting the value obtained by multiplying the radius (r) of cup 81 by the cosine of the cup CE angle (Θ) from the Y-coordinate (Y) of the intersection point (y = Y - r * cosΘ). When an arbitrary Θ is set, the center (x,y) of cup 81 corresponds one-to-one with the intersection point (X,Y). When the intersection point (X,Y) is changed, that is, when the amount of excavation by the bone excavation reamer 7 changes, the white line (Acetabular sourcil) indicating the pelvic load surface at the time of excavation changes, the center (x,y) of cup 81 also changes. In this way, when varying the intersection point (X, Y), for example, the cup CE angle (Θ) must exceed the minimum CE angle (10 degrees) (Θ > 10), which is a requirement for the implant. The processing unit 2 of the information processing device 1 may determine the range in which the intersection point (X, Y) can be varied within the range that satisfies this requirement (Θ > 10). In other words, the numerical value of Θ is not a fixed value (correct value), and the cup CE angle (Θ) that ensures cup fixation may be displayed on the monitor (display device 41) and used as one of the factors for determining the depth to be mined.

[0056] The processing unit 2 of the information processing device 1 identifies the region in which the center (x,y) of the cup 81 can be located, based on the range of variation of the determined intersection point (X,Y), using the above formula. The processing unit 2 of the information processing device 1 takes into account the radius (r) of the cup 81 and the center (x,y) of the cup 81 determined according to the varied intersection point (X,Y), and derives, as an example, the line of the edge that the outer edge of the bone excavation reamer 7 (the outer edge of the hemispherical tip that performs the excavation) must touch in order to satisfy the requirement (Θ>10) (the essential line). In deriving the essential line, the processing unit 2 of the information processing device 1 selects an arbitrary point (x1,y1) on the acetabular base of the pelvis before excavation with the bone excavation reamer 7, and sets the angle of the X-axis with respect to the line connecting the center (x,y) of the cup 81 and the arbitrary point as α. The coordinates of any point are (x+rcosα, y+rsinα). Then, the processing unit 2 of the information processing device 1 may derive the required line such that the point (x+rcosα, y+rsinα), which is the center (x,y) of the cup 81 corresponding to the changed intersection point (X,Y) and takes into account the radius (r) of the cup 81, is greater than the coordinates (x1,y1) of the arbitrary point (x+rcosα>x1 and y+rsinα>y2). In other words, the processing unit 2 of the information processing device 1 derives the required line as the edge of the range that can be the outer edge of the cup 81 by taking into account the range of (x',y') that the center (x,y) of the cup 81 that satisfies the condition equation "(x+rcosα>x1 and y+rsinα>y2" and, as an example, "cup CE angle; Θ>10", i.e., the radius (r) of the cup 81. The resulting target line corresponds to the inner line within the reachable area (the inner side that is closer to the user in the drilling direction of the bone excavation reamer 7).

[0057] The processing unit 2 of the information processing device 1 derives an upward limit line in the fluoroscopic image according to the center of the femoral head on the healthy side (S105). The processing unit 2 of the information processing device 1 recognizes the femoral head on the healthy side included in the fluoroscopic image and derives an upward limit line in the fluoroscopic image according to the center of the femoral head on the healthy side. The upward limit line indicates the limit of upward movement that the center of the femoral head on the affected side (the center of the femoral head ball of the artificial hip joint 8) can take relative to the center of the femoral head on the healthy side.

[0058] Figure 11 is an explanatory diagram relating to the upward limit line. In the illustration of this embodiment, four diagrams ((1) to (4)) are shown for reference. These diagrams are as follows: Diagram (1) shows a line parallel to the X-axis passing through y'=y2+10mm+cup 81 radius mm (cup 81 edge upward limit line). Diagram (2) shows a line parallel to the X-axis passing through y=y2+10mm (cup 81 center upward limit line). Diagram (3) shows a line parallel to the X-axis passing through the center of the healthy femoral head (x2,y2). Diagram (4) shows a pelvic reference line (X-axis) connecting the lower ends of the teardrop sigma. Patient K, who is to undergo treatment related to artificial hip joint 8, is generally expected to receive treatment related to artificial hip joint 8 (implant) on either the left or right leg. In this case, the side on which the treatment related to artificial hip joint 8 is performed is referred to as the affected side, and the healthy side on which the treatment related to artificial hip joint 8 is not performed is referred to as the healthy side.

[0059] The processing unit 2 of the information processing device 1 recognizes the shape of the healthy femoral head included in the acquired fluoroscopic image (X-ray image) using, for example, an object detection model such as YOLO or edge detection, and derives the center of the healthy femoral head by calculating the center of curvature from, for example, multiple points located on the outer edge, based on the outer edge which forms an arc shape on the femoral head. The processing unit 2 of the information processing device 1 derives an upward limit line (line segment (2)) of the cup 81 center (center of the femoral head of the implant on the affected side) that is within a predetermined value (for example, 10 mm) in the positive direction (upward direction) on the Y axis with respect to a line (line segment (3)) that passes through the derived healthy femoral head center (x2, y2) and is parallel to the X axis (line segment (4)). The upward limit line of the center of cup 81 (center of the femoral head of the implant on the affected side) serves as a reference line in determining the upward limit line of the periphery of cup 81 (line segment (1)), that is, the upward limit line (line segment (1)) in the reachable area (strike zone).

[0060] The center of the bone on the affected side where the treatment for the artificial hip joint 8 is performed corresponds to the center (x, y) of the cup 81 to be placed. The Y-axis indicates the superior direction in the human body, and the positive direction on the Y-axis indicates upward. The Y coordinate (y) of the center (x, y) of the cup 81 to be placed is set such that it does not exceed, for example, 10 mm upward (y < y2 + 10 [mm]) with respect to the Y coordinate (y2) of the center of the bone on the healthy side. Therefore, the processing unit 2 of the information processing apparatus 1 passes through a point moved upward (along the positive direction of the Y-axis) by the upward limit value (for example, 10 mm) with respect to the Y coordinate (y2) of the center of the bone on the healthy side, and derives an upward limit line (line segment (2)) of the center of the cup 81 (the center of the bone of the implant on the affected side) that is perpendicular to the Y coordinate. Further, the processing unit 2 of the information processing apparatus 1 is parallel to the upward limit line (line segment (2)) of the center of the cup 81 (the center of the bone of the implant on the affected side) (that is, parallel to the X-axis), and is located upward (in the positive direction on the Y-axis) by the radius of the cup 81. The upward limit line of the cup 81 edge (line segment (1)), that is, the upward limit line (line segment (1)) in the reach area (strike zone) is determined (derived). By deriving the upward limit line (line segment (1): upward limit line of the cup 81 edge) in the reach area (strike zone) based on the center of the bone on the healthy side in this way, a guide (the upper limit line in the excavation of the pelvis) when the center of the bone on the affected side (the center of the femoral head of the artificial hip joint 8) is placed can be provided to doctors and the like.

[0061] The processing unit 2 of the information processing apparatus 1 derives the reach area (strike zone) based on the derived inner plate line, must-reach line, and upward limit line (S106). Alternatively, the processing unit 2 of the information processing apparatus 1 may derive the reach area (strike zone) based on the derived inner plate line, the position information of the acetabular cup (anterior and posterior wall area information, central area information), and the bone coverage rate (BCI, three-dimensional bone coverage rate). The processing unit 2 of the information processing apparatus 1 derives the area surrounded by these lines as the reach area (strike zone) based on the derived upward limit line, inner plate line, and must-reach line.

[0062] Figure 12 is an explanatory diagram regarding the reachable area (strike zone). This reachable area indicates the region in which the stable fixation of the cup 81 is ensured if a part of the cup 81 is in contact with the area when the cup 81 is placed. Within the reachable area, the upward limit line and the inner plate line indicate the limit lines that must not be exceeded when drilling with the bone drilling reamer 7. By superimposing the thus derived reachable area onto the fluoroscopic image, the positional relationship between the current position of the bone drilling reamer 7 and the region that the bone drilling reamer 7 should reach by drilling (the reachable area) can be provided to the physician in real time by the bone drilling reamer 7 included in the real-time displayed fluoroscopic image.

[0063] The processing unit 2 of the information processing device 1 derives various measured values, target values, and the difference between them at the present time (S107). The processing unit 2 of the information processing device 1 derives the measured values ​​of the cup 81 that are expected to be positioned after the pelvis has been shaved by the bone excavation reamer 7, using fluoroscopic images. The target CE angle, target bone coverage rate, and target distance between the outer edge of the cup 81 and the inner plate line are set in advance as target values ​​and are stored in the storage unit 3 of the information processing device 1 as various values ​​determined in the preoperative plan when performing treatment related to the artificial hip joint 8. The processing unit 2 of the information processing device 1 obtains these various values ​​determined in the preoperative plan by referring to the storage unit 3.

[0064] The processing unit 2 of the information processing device 1 uses the acquired fluoroscopic image to determine the position and inclination of the cup 81 when the pelvis has been shaved by the bone-excavating reamer 7, based on the current position and inclination of the bone-excavating reamer 7 (the outer edge of the hemispherical tip that performs the excavation) in the pelvis of patient K. This inclination includes, for example, the anteversion angle and the abduction angle of the cup 81.

[0065] Figure 13 is an explanatory diagram regarding the anterior tilt angle and abduction angle of the cup 81 (assumed placement when the pelvis has been shaved by the bone-excavating reamer 7). The processing unit 2 of the information processing device 1 identifies three points (A, A', B) of an ellipse using shape recognition processing. In this case, if the angle between A-A' and A'-B in the fluoroscopic image is X, the anterior tilt angle is indicated by "sin^(-1)*tanX" or the like. The processing unit 2 may also recognize the shape and current position of the bone-excavating reamer 7 (outer edge of the hemispherical tip that performs excavation) corresponding to the assumed placement of the cup 81 in the fluoroscopic image by calculation using the image coordinate system in the fluoroscopic image, and calculate the anterior tilt angle and abduction angle.

[0066] The processing unit 2 of the information processing device 1 may derive the target forward tilt angle and target abduction angle based on a preset target CE angle. Alternatively, the processing unit 2 of the information processing device 1 may derive the target forward tilt angle and target abduction angle by referring to a table (target CE angle table) in which the values ​​of the target forward tilt angle and target abduction angle are associated with each value of the target CE angle. Various lookup tables, such as the target CE angle table, that the processing unit 2 of the information processing device 1 refers to when performing various calculations are stored in the storage unit 3.

[0067] The processing unit 2 of the information processing device 1 outputs various derived support information (inner plate lines, reachable area, difference between measured values ​​and target values) superimposed on and accompanying the perspective image (S108). The support information includes, for example, inner plate lines, reachable area, and the difference between measured values ​​and target values. The processing unit 2 of the information processing device 1 superimposes the reachable area, including the derived inner plate lines, onto the perspective image and outputs it to the display device 41. The processing unit 2 of the information processing device 1 further outputs various measured values, target values, and the difference between these measured values ​​and target values ​​accompanying the perspective image to the display device 41. When outputting this information, the processing unit 2 of the information processing device 1 may generate and output screen data that constitutes a display screen (support information superimposed screen). The processing unit 2 of the information processing device 1 compares the derived measured values ​​(cup CE angle, bony coverage, measured distance between the outer edge of cup 81 and the inner plate line, anterior tilt angle, abduction angle) with the corresponding target values ​​(target CE angle, target bony coverage, target distance between the outer edge of cup 81 and the inner plate line, target anterior tilt angle, target abduction angle) and displays them together with the fluoroscopic image (for example, in a sub-screen or separate frame). This allows physicians and others to efficiently grasp the differences between various measured values ​​and target values ​​at the present time.

[0068] Figure 14 is an explanatory diagram illustrating a support information display screen (support information overlay screen). The processing unit 2 of the information processing device 1 generates screen data constituting the support information overlay screen as a result of the various processes described above, and outputs the screen data to the display device 41. The support information overlay screen includes a fluoroscopic image display area, a acetabular position relationship display area, and a support information display area.

[0069] The fluoroscopic image display area shows a fluoroscopic image (X-ray image) of patient K's pelvis, captured in real time during surgery while the pelvis is being cut by a bone-excavating reamer 7. The reach area, including the inner plate line, is superimposed on the fluoroscopic image.

[0070] The acetabular positional relationship display area displays a fluoroscopic image (X-ray image) of patient K's pelvis, captured in real time during surgery when the pelvis is being cut by a bone-excavating reamer 7. The anterior and posterior walls of the acetabulum are superimposed on the fluoroscopic image. Alternatively, the area where the center of the cup 81 is located when the cup 81 is placed in contact with the anterior and posterior walls of the acetabulum is superimposed on the fluoroscopic image. Furthermore, the load-bearing surface area may also be superimposed on the fluoroscopic image.

[0071] The support information display area displays, in list format, the anterior tilt angle, abduction angle, cup CE angle, bony coverage rate, and remaining distance from the inner plate line of the cup 81, assuming the pelvis has been shaved by the bone-excavating reamer 7. For these anterior tilt angle, abduction angle, cup CE angle, bony coverage rate, and remaining distance from the inner plate line of the cup 81, the measured values ​​calculated based on the current position and inclination of the bone-excavating reamer 7 identified by shape recognition processing in the fluoroscopic image, the values ​​determined in the preoperative plan (target values), and the difference between the measured values ​​and the target values ​​are displayed in list format.

[0072] Figure 15 is an explanatory diagram illustrating a support information display screen (distortion correction screen). The processing unit 2 of the information processing device 1 generates screen data constituting the distortion correction screen as a result of the various processes described above, and outputs the screen data to the display device 41. The distortion correction screen includes a display area before distortion correction and a display area after distortion correction. In this way, the distortion correction screen displays a comparison between the fluoroscopic image before distortion correction (with support information superimposed) and the fluoroscopic image after distortion correction (with support information superimposed), so that it can present information to doctors and others showing the before and after of distortion correction for the target area (strike zone), and can provide useful information for the doctor when performing a procedure.

[0073] The distortion correction screen may be output on a separate screen from the aforementioned support information overlay screen, or it may be included in the support information overlay screen. When the distortion correction screen is output on a separate screen from the support information overlay screen, the display device 41 that displays the distortion correction screen and the display device 41 that displays the support information overlay screen may be separate display devices 41. In this case, the information processing device 1 will be connected to two display devices 41: one for displaying the distortion correction screen and another for displaying the support information overlay screen.

[0074] The pre-distortion correction display area displays the image shown in the perspective image display area of ​​the aforementioned support information superimposed screen (a perspective image with the reachable area superimposed). Furthermore, the image displayed in the pre-distortion correction display area (pre-correction image) shows a grid of auxiliary lines consisting of multiple straight lines used when performing distortion correction.

[0075] The distortion-corrected display area shows the distortion-corrected image (a perspective image with the target area superimposed). Furthermore, the image displayed in the distortion-corrected display area (the corrected image) shows the auxiliary lines used during the distortion correction process in a curved form corresponding to the correction. As a result, the target area (strike zone) is also displayed with a changed shape (the area in the pelvis) according to the distortion correction.

[0076] The processing unit 2 of the information processing device 1 may, for example, perform distortion correction by attaching a template containing straight lines below the fluoroscopic image (image before correction). In this case, the processing unit 2 of the information processing device 1 is a device that converts X-rays that have passed through the human body into a digital image, and may use a flat panel with a distortion correction function.

[0077] Alternatively, the processing unit 2 of the information processing device 1 may perform distortion correction processing on the fluoroscopic image (pre-correction image) according to the distortion aberration determined based on the X-ray imaging characteristics of the X-ray machine 62, etc. The X-ray machine 62 that takes fluoroscopic images (X-ray images) has various characteristics depending on the model or type, and the distortion aberration corresponding to the amount of distortion in the captured fluoroscopic image (X-ray image) can also be specified. The storage unit 3 of the information processing device 1 stores parameters such as distortion aberration corresponding to each of the various X-ray machines 62, and the processing unit 2 of the information processing device 1 may perform distortion correction using parameters such as distortion aberration corresponding to the X-ray machine 62, etc. In this case, it is not necessary to use a flat panel, which is a dedicated device that performs distortion correction by hardware processing, and the distortion-corrected fluoroscopic image can be displayed using a relatively inexpensive display (display device 41 that does not have a hardware-processing distortion correction function).

[0078] According to this embodiment, the processing unit 2 of the information processing device 1 acquires a fluoroscopic image (X-ray image) of patient K who is undergoing treatment related to the artificial hip joint 8. The fluoroscopic image is, for example, an X-ray image. As part of the treatment related to the artificial hip joint 8, when a bone reamer 7 is used to form a bone cavity (installation space) in the pelvis that will serve as a receptacle for installing the cup 81 of the artificial hip joint 8 (excavating the pelvis), the fluoroscopic image (X-ray image) is captured in real time as a moving image, and the processing unit 2 of the information processing device 1 sequentially acquires the fluoroscopic image (X-ray image) that has been captured in real time. The processing unit 2 of the information processing device 1 recognizes (derives) an inner plate line indicating the maximum cutting point (limit depth) when cutting the pelvis to place the cup 81 containing the artificial hip joint 8, and a required line (line to be exceeded) indicating the minimum cutting point when cutting the pelvis to place the cup 81, in the acquired fluoroscopic image of patient K's pelvis. Alternatively, the processing unit 2 of the information processing device 1 recognizes (derives) acetabular positional relationship information (anterior-posterior wall region information, central region information) that indicates the contact relationship between the anterior and posterior walls of the acetabulum and the cup 81 in the pelvis of patient K, so that the cup 81 can obtain sufficient initial fixation. Information regarding the inner plate line is derived based on the positional information of the inner plate output from the learning model 101 by inputting the acquired fluoroscopic image (X-ray image) into the learning model 101. That is, the positional information of the inner plate from the learning model 101 may correspond to the inner plate line itself, or the positional information of the inner plate from the learning model 101 may be a plurality of points (point cloud) that constitute the inner plate line, and the processing unit 2 of the information processing device 1 may generate the inner plate line using the point cloud. Furthermore, the processing unit 2 of the information processing device 1 derives the cup CE angle or bone coverage ratio (BCI, 3D bone coverage ratio) when the placement of the cup 81 is assumed to be in a state where the pelvis has been shaved by the bone excavation reamer 7 (in real time). The processing unit 2 of the information processing device 1 superimposes the recognized (derived) inner plate lines and required lines onto the fluoroscopic image, and also displays the derived cup CE angle attached to the fluoroscopic image, for example, on a sub-screen, thereby providing useful support information to a physician performing a procedure on patient K regarding the artificial hip joint 8.The processing unit 2 of the information processing device 1 recognizes the shape of the outer edge (the outer edge of the hemispherical tip that performs drilling) of the bone drilling reamer 7 included in the fluoroscopic image (X-ray image) using, for example, an object detection model such as YOLO or edge detection. The processing unit 2 of the information processing device 1 considers the outer edge of the bone drilling reamer 7 as the outer edge of the cup 81 and derives the cup CE angle when the cup 81 is assumed to be positioned in the state where the pelvis has been drilled by the bone drilling reamer 7. The processing unit 2 of the information processing device 1 derives the cup CE angle as the angle formed by the line connecting the intersection of the outer edge of the cup 81 and the white line (Acetabular sourcil) on the pelvic load surface (COR: Center of Rotation) of the cup 81 assumed to be positioned in the state where the pelvis has been drilled by the bone drilling reamer 7, and the line connecting the lower ends of the left and right teardrop marks on the pelvis, which is perpendicular to the pelvic reference line (Y-axis). The processing unit 2 of the information processing device 1 recognizes the shape of the lower ends of the left and right teardrops included in the fluoroscopic image (X-ray image) using, for example, an object detection model such as YOLO or edge detection, and identifies a pelvic reference line that passes through these lower ends of the left and right teardrops. The processing unit 2 of the information processing device 1 identifies a vertical line that is perpendicular to the identified pelvic reference line and passes through the center (COR) of the cup 81, which is assumed to be positioned in a state where the pelvis has been shaved by the bone excavation reamer 7. The processing unit 2 of the information processing device 1 derives the cup CE angle by calculating the angle between the vertical line passing through the center (COR) of the cup 81 and the line connecting the intersection point of the outer edge of the cup 81 and the white line (Acetabular sourcil) of the pelvic load surface. The processing unit 2 of the information processing device 1 acquires the fluoroscopic image captured in real time and derives the cup CE angle based on the outer edge of the bone excavation reamer 7 included in the acquired fluoroscopic image. This allows the physician to be provided with the cup CE angle if the cup 81 were placed at the current time, in accordance with the current position of the bone excavation reamer 7, i.e., the excavation state by the bone excavation reamer 7. In other words, an intraoperative support system S can be configured that enables real-time evaluation of cup placement and coverage during surgery.

[0079] According to this embodiment, the storage unit 3 of the information processing device 1 stores an actual file of a learning model 101 (inner plate line model) that has been trained to output inner plate position information (landmarks) when a fluoroscopic image (X-ray image) is input. The processing unit 2 of the information processing device 1 inputs the acquired fluoroscopic image (X-ray image) to the learning model 101 (inner plate line model), and the learning model 101 (inner plate line model) outputs the inner plate position information superimposed on the fluoroscopic image. Each of the inner plate position information is output as a point (landmark), and by outputting multiple points (landmarks) on the fluoroscopic image, the inner plate line may be identified by forming (connecting) lines between these multiple points (landmarks). By using the learning model 101 (inner plate line model) that has been trained to output inner plate position information (landmarks) when a fluoroscopic image is input in this way, the inner plate line indicating the maximum cutting point when cutting the pelvis to position the cup 81 containing the artificial hip joint 8 can be efficiently identified (derived). The processing unit 2 of the information processing device 1 displays (outputs) the inner plate line, derived using the learning model 101 based on the fluoroscopic image (X-ray image), superimposed on the fluoroscopic image (X-ray image). The fluoroscopic image (X-ray image) displayed in real time during surgery shows the bone excavation reamer 7 at the current moment, and the inner plate line is superimposed on it. This effectively presents the physician or other operator of the bone excavation reamer 7 with the inner plate line (excavation limit line) that marks the maximum excavation point when cutting the pelvis, i.e., the limit point that must not be exceeded.

[0080] According to this embodiment, the learning model 101 is generated by training it using training data. Therefore, the training data used is data in which the positional information of the inner plate is annotated to fluoroscopic images (X-ray images) of a patient or subject different from patient K, who will undergo treatment related to the artificial hip joint 8 in this case. In this case, the fluoroscopic images (X-ray images) used in the training data include pseudo-fluoroscopic images (DRR-derived fluoroscopic images [DRR-F]) generated from DRR images (Digital Reconstructed Radiograph: CT reconstruction simulation images) converted or reconstructed using multiple tomographic images (CT images) taken of the subject or the like. The processing unit 2 of the information processing device 1 can align the DRR images converted from tomographic images (CT images) and the pseudo-fluoroscopic images (DRR-F) generated from the DRR images by applying the same internal body coordinate system, and can perform various calculations on these images (CT images, DRR images, and pseudo-fluoroscopic images). The simulated fluoroscopic images (DRR-F) show the pelvis from multiple different fluoroscopic angles, such as the front (anterior pelvis) and lateral (lateral pelvis). These simulated fluoroscopic images are annotated with an inner platen line (or a point cloud constituting the inner platen line) based on the positional information of the inner platen defined in multiple tomographic images (CT images). In other words, simulated fluoroscopic images (DRR-F) from many directions consisting of multiple different fluoroscopic angles are created, and the inner platen line is annotated not only for the front (anterior pelvis) but also for simulated fluoroscopic images (DRR-F) that include angles other than the front of the pelvis. Therefore, even when the intraoperative fluoroscopic image is not of the front of the pelvis, the inner platen line can be accurately displayed. The simulated fluoroscopic images (DRR-F) correspond to the problem data in the training data, and the inner platen line annotated on the simulated fluoroscopic images (DRR-F) corresponds to the answer data in the training data. The response data annotated to the pseudo-perspective image (DRR-F) is not limited to the inner plate line, but may also be a diagram of the outer edge of the cup 81 at the installation position when the cup 81 is installed, derived based on the inner plate line, the required line and the cup CE angle.In other words, the training data may be a pseudo-fluoroscopic image (DRR-F) to which the outer edge of the cup 81 at the installation position when the cup 81 is placed is annotated. By generating a pseudo-fluoroscopic image using a DRR image converted from a tomographic image (CT image) in this way, it is possible to obtain a pseudo-fluoroscopic image that is less blurry and relatively clear compared to, for example, an X-ray image directly and pseudo-created from a CT image. When generating a pseudo-fluoroscopic image (DRR-F) from a DRR image, the tomographic image (CT image) may be adjusted or corrected so that the pseudo-fluoroscopic image (DRR-F) is a complete frontal view of the pelvis, and the DRR image may be reconstructed. By generating the pseudo-fluoroscopic image (DRR-F) in such a way that the pelvis included in the pseudo-fluoroscopic image (DRR-F) is perfectly frontal, angle correction of the X-ray image relative to the pelvis (frontal view of the pelvis) is performed, and the fluoroscopic image (X-ray image) of patient K acquired during surgery and the pseudo-fluoroscopic image (DRR-F) can be accurately aligned or superimposed. Multiple pseudo-fluoroscopic images (DRR-F) with different fluoroscopic angles, i.e., different angles of X-ray irradiation to the subject, may be generated from the DRR image converted from the tomographic image (CT image). In other words, since multiple pseudo-fluoroscopic images (DRR-F) with different fluoroscopic angles can be generated from the same DRR image, a large number of pseudo-fluoroscopic images (DRR-F) can be generated and training data can be increased even when the number of subjects in the tomographic image (CT image) is relatively small. Thus, in the generation stage (learning process) of the learning model 101, tomographic images (CT images) taken of the subject are required. However, when the learning model 101 is put into operation, that is, when the learning model 101 is used to perform a procedure on patient K related to the artificial hip joint 8, it is not necessary to take CT images of patient K (preoperative CT imaging). Depending on the size of the medical institution where the procedure on the artificial hip joint 8 is performed, it is conceivable that they may have an X-ray imaging device for taking fluoroscopic images (X-ray images) but not a CT scanner 61. In such cases, by using the learning model 101 which uses only fluoroscopic images (X-ray images) as input data, the availability of the intraoperative support system S can be improved.In other words, by using the intraoperative support system S, when performing a procedure on patient K regarding the artificial hip joint 8, it is possible to display the cup placement angle of the artificial hip joint 8, display the cup placement position information, and correct the angle of the X-ray image relative to the pelvis (front view of the pelvis) using only intraoperative fluoroscopic images (X-ray images).

[0081] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of the invention is indicated by the claims and not in the sense described above, and all modifications within the sense and scope equivalent to the claims are intended to be included.

[0082] With respect to the multiple claims described in the claims, they may be combined with each other regardless of the form of reference. Multiple dependent claims that depend on multiple claims may be described in the claims. Multiple dependent claims that depend on multiple dependent claims may be described. Even if multiple dependent claims that depend on multiple dependent claims are not described, this does not limit the description of multiple dependent claims that depend on multiple dependent claims.

[0083] S Intraoperative support system K Patient 1 Information processing device 2 Processing unit 3 Memory unit M Recording medium P Program (Program product) 4 Input / output I / F 41 Display device 5 Communication unit 101 Learning model (Position information model) 102 Second learning model (Bone coverage model) 61 CT scanner (Tomographic imaging device) 62 X-ray scanner (Fluoroscopy imaging device) 7 Bone drilling reamer 8 Artificial hip joint 81 Cup

Claims

1. A program that causes a computer to acquire fluoroscopic images of a patient undergoing a procedure related to an artificial hip joint including a cup, and to execute a process to derive positional information of the anterior wall and posterior wall of the acetabulum of the patient by inputting the acquired fluoroscopic images into a learning model that has been trained to output positional information of the anterior wall and posterior wall of the acetabulum of the patient when the fluoroscopic images are input.

2. The program according to claim 1, which recognizes acetabular positional relationship information relating to the positional relationship between the cup and the acetabulum when the cup is placed in the patient's pelvis in the acquired fluoroscopic image, based on the derived positional information of the anterior and posterior walls of the acetabulum.

3. The program according to claim 2, which derives the bone coverage rate when the placement of the cup is assumed to be in a state in which the patient's pelvis has been shaved with a bone-excavating reamer used in the procedure for the artificial hip joint.

4. The program according to claim 3, which superimposes the recognized acetabular positional relationship information onto the fluoroscopic image and displays the derived bony coverage rate associated with the fluoroscopic image.

5. The program according to claim 4, wherein the acetabular positional relationship information includes anterior and posterior wall region information indicating the regions of the anterior and posterior walls of the acetabular fold that come into contact with the front and rear of the cup when the cup is placed.

6. The program according to claim 4, wherein the acetabular positional relationship information includes central region information indicating the region where the center of the cup is located when the cup is placed in contact with the anterior and posterior walls of the acetabular lid.

7. The program according to claim 4, wherein the bone coverage rate includes at least one of BCI (Bone coverage index), cup CE angle, and three-dimensional bone coverage rate.

8. The program according to claim 4, wherein the training data for the learning model is generated based on pseudo-fluoroscopic images generated from tomographic images of a subject different from the patient.

9. The program according to claim 8, wherein the pseudo-fluoroscopic image includes a plurality of pseudo-fluoroscopic images generated at a plurality of different fluoroscopic angles with respect to the DRR image converted from the tomographic image.

10. The program according to claim 4, which derives an upward limit line relative to the center of the femoral head on the affected side to which the artificial hip joint procedure is performed, based on the center of the femoral head on the healthy side that is not subjected to the artificial hip joint procedure, derives a target area based on the derived upward limit line, the maximum cutting point when cutting the pelvis, and the acetabular position relationship information, and displays the derived target area superimposed on the fluoroscopic image.

11. The program according to claim 4, which derives a minimum cutting line indicating the point at which the pelvis of the patient is cut in order to position the cup, based on a predetermined target CE angle or target bony coverage ratio for stabilizing the cup to be positioned, the intersection point of the outer edge of the cup which serves as a reference when ensuring the target CE angle or target bony coverage ratio and the inner edge of the load-bearing surface in the patient's pelvis, and the radius of the cup.

12. The program according to claim 4, which outputs a comparison between the derived bone coverage rate and a target CE angle or target bone coverage rate set in advance as a target value; derives the measured distance between the outer edge of the cup and the point of maximum cutting when cutting the pelvis, assuming the placement of the cup when the patient's pelvis is cut by the bone excavation reamer; derives the target distance between the outer edge of the cup and the point of maximum cutting when cutting the pelvis, set in advance as a target value; and outputs a comparison between the derived measured distance and the target distance.

13. The program according to claim 4, which derives the anterior tilt angle and abduction angle of the cup when the placement of the cup is assumed to be in a state in which the patient's pelvis has been shaved by the bone-excavating reamer, and outputs the derived anterior tilt angle and abduction angle.

14. The program according to claim 12, which derives a target forward tilt angle and a target abduction angle according to the target CE angle or target bony coverage, and outputs a combination of the derived target forward tilt angle and the target abduction angle.

15. The program according to claim 14, which outputs a comparison between the derived forward tilt angle and abduction angle of the cup and the target forward tilt angle and the target abduction angle.

16. The program according to claim 8, which obtains the current position of the bone drilling reamer in the patient's pelvis based on the acquired fluoroscopic image, extracts a tomographic image containing the current position of the bone drilling reamer from among a plurality of tomographic images of the patient's pelvis, and outputs a diagram of the outer edge of the cup, assuming the cup is positioned in a state where the patient's pelvis has been drilled by the bone drilling reamer, superimposed on the extracted tomographic image.

17. An information processing method that causes a computer to acquire fluoroscopic images of a patient undergoing artificial hip joint treatment, and to execute a process to derive positional information of the anterior wall of the acetabulum and the posterior wall of the acetabulum by inputting the acquired fluoroscopic images into a learning model that has been trained to output positional information of the medial plate, the anterior wall of the acetabulum, and the posterior wall of the acetabulum when the fluoroscopic images are input.

18. An information processing device comprising: an acquisition unit for acquiring fluoroscopic images of a patient undergoing artificial hip joint surgery; and a position information derivation unit for deriving position information of the anterior wall of the acetabulum and the posterior wall of the acetabulum of the patient by inputting the acquired fluoroscopic images into a learning model that has been trained to output position information of the inner plate, the anterior wall of the acetabulum, and the posterior wall of the acetabulum when a fluoroscopic image is input.

19. A method for generating a learning model that acquires training data including fluoroscopic images of a patient undergoing artificial hip joint surgery and positional information of the patient's inner plate, and generates a learning model that outputs positional information of the anterior wall and posterior wall of the acetabulum when a fluoroscopic image is input, based on the acquired training data, wherein the fluoroscopic image includes a plurality of pseudo-fluoroscopic images generated at a plurality of different fluoroscopic angles for a DRR image converted from a tomographic image of the patient.

Images