Information processing method, method for generating trained model, information processing system, and program

Abstract

One aspect of the present invention provides an information processing method comprising an acquisition step, a generation step, a conversion step, and an output step. In the acquisition step, a sensor point cloud layer defined as a layer of a point cloud representing the arrangement of a plurality of sensors, a target point cloud layer defined as a layer of a point cloud representing a region to be sensed, and measurement data measured by each sensor are acquired, wherein a physical distance is defined to be present between the sensor point cloud layer and the target point cloud layer. In the generation step, at least one intermediate point cloud layer defined as a layer of a point cloud positioned between the sensor point cloud layer and the target point cloud layer is generated. In the conversion step, each piece of measurement data is propagated stepwise as a feature amount of each point included in the intermediate point cloud layer from the sensor point cloud layer toward the target point cloud layer via the intermediate point cloud layer. In the conversion step, each feature amount propagated to the target point cloud layer is converted into an attribute value of each point in the target point cloud layer. In the output step, the attribute value is output.

Description

Information processing method, method for generating trained models, information processing system and program

[0001] This invention relates to an information processing method, a method for generating a trained model, an information processing system, and a program.

[0002] Patent Document 1 discloses a system for diagnosing arrhythmias in patients.

[0003] The system for diagnosing arrhythmias in this patient includes a diagnostic catheter for insertion into the patient's heart, configured to record the patient's anatomical and electrical activity data, and a processing unit. The processing unit is configured to receive the recorded electrical activity data and correlate the electrical activity data with the anatomical data. The processing unit includes an algorithm configured to analyze the electrical activity at locations associated with the anatomical data.

[0004] Japanese Patent Publication No. 2023-182931

[0005] However, the technology disclosed in Patent Document 1 still required the catheter sensor to be in contact with the surface of cardiac tissue. This need for contact with the cardiac tissue surface limited the range that could be measured by the sensor, thus increasing the measurement time for diagnosis. Therefore, there was a need for a less invasive method to reduce the burden on patients in various measurements, including the diagnosis of arrhythmias.

[0006] Incidentally, in order to accurately acquire information from sensing targets such as the heart, it is conceivable to arrange sensors densely. However, when placing sensors on a catheter, for example, there is a limit to the number of sensors that can be placed, so it was necessary to weigh the accuracy of the measurement against the field of view when arranging the sensors. Such problems were difficult to solve in measurements that required image reconstruction.

[0007] In view of the above circumstances, the present invention aims to provide an information processing method, a method for generating a trained model, an information processing system, and a program, etc., that can solve at least one of the following (1) and (2) while ensuring measurement accuracy in measurements requiring image reconstruction: (1) To improve the degree of freedom of the physical distance between the sensor and the sensing object compared to conventional methods. (2) To enable sensors to be arranged more sparsely than conventional methods.

[0008] According to one aspect of the present invention, an information processing method is provided, comprising an acquisition step, a generation step, a conversion step, and an output step, wherein in the acquisition step, a sensor point cloud layer defined as a layer of point clouds representing the arrangement of a plurality of sensors, a target point cloud layer defined as a layer of point clouds representing the area to be sensed, and measurement data measured by each sensor are acquired, and a physical distance is defined between the sensor point cloud layer and the target point cloud layer, in the generation step, one or more intermediate point cloud layers defined as layers of point clouds located between the sensor point cloud layer and the target point cloud layer are generated, in the conversion step, each measurement data is propagated stepwise from the sensor point cloud layer to the target point cloud layer via the intermediate point cloud layer as a feature quantity of each point included in the intermediate point cloud layer, in the conversion step, each feature quantity propagated to the target point cloud layer is converted into an attribute value of each point in the target point cloud layer, and in the output step, the attribute value is output.

[0009] When a physical distance is defined between the sensor point cloud layer and the target point cloud layer, the correlation between the measurement data measured by each sensor and the attribute values ​​of the sensing target weakens according to that physical distance. Specifically, taking the measurement of cardiac membrane potential as an example, as the distance between the sensor and the cardiac tissue surface increases, the amplitude of the measurement signal is attenuated, and the state of a wide range of tissues affects the measurement signal, making it difficult to reconstruct the image accurately.

[0010] Therefore, the method in the above embodiment defines an intermediate point cloud layer between the sensor point cloud layer and the target point cloud layer, thereby converting the measurement data into attribute values ​​via the intermediate point cloud layer. In other words, the concept of an intermediate point cloud layer can be defined by defining a physical distance between the sensor point cloud layer and the target point cloud layer. Furthermore, the method in the above embodiment, by defining an intermediate point cloud layer, converts the measurement data into another feature quantity via the intermediate point cloud layer, and then converts it back to the final attribute value via that feature quantity. In other words, by performing the conversion from measurement data to attribute values ​​in multiple stages, the influence of the distance between the sensor and the tissue surface is reduced, thus ensuring better correlation than performing data conversion in a single stage.

[0011] According to the above embodiment, measurement accuracy is ensured even when the sensor density per unit area is somewhat sparse. Therefore, for example, when placing sensors on a catheter, the sensors can be sparsely arranged, and the removed sensors can be placed in other locations to secure the measurement field of view. Furthermore, according to the above embodiment, measurement accuracy is ensured even when the sensors do not come into contact with the sensing object, so the attribute values ​​of the sensing object can be imaged with greater accuracy than before.

[0012] Furthermore, by using the method of this embodiment, sensors can be arranged more sparsely than in the conventional method, thereby reducing the load when processing sensor signals. Also, by using the method of this embodiment, the degree of freedom in the physical distance between the sensor and the sensing object is improved compared to the conventional method, so that the measurement time can be shortened and the power consumption of the computer can be reduced.Therefore, the method of this embodiment can improve the functions of the computer to achieve at least one of the following (1) to (4): (1) The computer's processing speed can be increased. (2) The computer's power consumption can be reduced. (3) The computer's communication speed can be increased. (4) The resources saved in the computer can be used for other core functions.

[0013] The means for solving the above problems makes it possible to solve at least one of the following (1) and (2) in measurements that require image reconstruction, while ensuring the accuracy of the measurement: (1) improving the degree of freedom of the physical distance between the sensor and the sensing object compared to conventional methods; and (2) arranging the sensors more sparsely than conventional methods.

[0014] This is a diagram showing the configuration of the information processing system 100. This is a block diagram showing the hardware configuration of the information processing device 200. This is a block diagram showing the hardware configuration of the terminal 300. This is a block diagram showing the functions realized by the information processing device 200 (control unit 210). This is an activity diagram showing the flow of the information processing method executed by the information processing device 200. This is an activity diagram showing the flow of the information processing method executed by the information processing device 200. This is a diagram showing an example of a sensor point cloud layer, a target point cloud layer, and measurement data. This is a conceptual diagram corresponding to activity A140. This is a conceptual diagram corresponding to activities A150 and A152. This is a diagram showing the state in which the feature quantities of each point included in the intermediate point cloud layer have been converted into attribute values. This is an activity diagram showing the flow of the method for generating a trained model executed by the terminal 300. This is a diagram showing an example of a dataset created in the method for generating a trained model. This is a diagram showing the accuracy of the prediction results by the trained model generated in Section 4-1. This is a diagram showing the membrane potential mapping results by the trained model generated in Section 4-1. This is a diagram showing the process of generating an intermediate point cloud layer by the trained model of Example 2. This is a diagram showing an overview of the training dataset used when generating the trained model of Example 2. This figure shows a comparison between propagating features via an intermediate point cloud layer and directly propagating features without passing through an intermediate point cloud layer, using the trained model of Example 2. This figure illustrates the time shift of the ultrasonic signal. This figure illustrates the process performed when extracting features from each point. This figure illustrates the process performed when interpolating new points. This figure illustrates an overview of the experimental example in this embodiment. This figure illustrates a method for converting the estimated virtual signal into an image. This figure shows an example of sensor placement in the training dataset. This figure shows the conditions of the experimental example in this embodiment. This figure shows the estimated signal and estimated image when two sound sources are placed. This figure shows the estimated signal and estimated image when four sound sources are placed. This figure shows the results of comparing the image quality of the estimated image by the proposed method with the image quality of the estimated image by the conventional method (Delay & Sum method).

[0015] The first and second embodiments of the present invention will be described below with reference to the drawings. The various features shown in the embodiments below can be combined with each other. The first embodiment will be described in Sections 1 to 5, and the second embodiment will be described in Section 6.

[0016] Incidentally, the program for realizing the software appearing in one embodiment may be provided as a computer-readable non-transitor-readable medium, or it may be provided so that it can be downloaded from an external server, or it may be provided so that the program is launched on an external computer and its functions are realized on a client terminal (so-called cloud computing).

[0017] Furthermore, in various information processing according to one embodiment, an input and an output corresponding to the input can be realized. Here, as long as an output is obtained as a result of the input, the form of the information referenced in such information processing (hereinafter referred to as "reference information") is not limited. The reference information may be, for example, rule-based information such as a database, a lookup table, or a predetermined function (including a decision formula such as a regression equation constructed by a statistical method), or a trained model that has been pre-trained to learn the correlation between input and output, or a generative AI such as a large-scale language model that can output a desired result by inputting a prompt (these models include parameters that construct the correlation relationship between input and output) or a visual language model.

[0018] Furthermore, in one embodiment, "part" may include, for example, hardware resources implemented by a circuit in a broad sense, and the information processing of software that can be specifically realized by these hardware resources. Also, in one embodiment, various types of information are handled, and this information can be represented, for example, by the physical values ​​of signal values ​​representing voltage and current, the high or low values ​​of signal values ​​as a set of binary bits composed of 0s or 1s, or by quantum superposition (so-called qubits), and communication and calculations can be performed on a circuit in a broad sense.

[0019] Furthermore, a circuit in a broad sense is a circuit realized by combining at least an appropriate combination of circuits, circuits, processors, and memory. The processor may be a general-purpose processor or a dedicated circuit. In other words, this includes application-specific integrated circuits (ASICs), programmable logic devices (for example, simple programmable logic devices (SPLDs), complex programmable logic devices (CPLDs), and field programmable gate arrays (FPGAs)), etc.

[0020] 1. Hardware Configuration Section 1 describes the hardware configuration of this embodiment.

[0021] 1-1. Information Processing System 100 Figure 1 is a configuration diagram representing the information processing system 100. The information processing system 100 comprises an information processing device 200 and a terminal 300, which are connected via a network. These components will be explained further. Here, the system exemplified in the information processing system 100 consists of one or more devices or components. Therefore, for example, even the information processing device 200 alone can be a system exemplified in the information processing system 100.

[0022] 1-2. Information Processing Device 200 Figure 2 is a block diagram showing the hardware configuration of the information processing device 200. The information processing device 200 includes a control unit 210, a storage unit 220, and a communication unit 250, and these components are electrically connected within the information processing device 200 via a communication bus 260. Each component will be described further.

[0023] The control unit 210 performs processing and control of the overall operation related to the information processing device 200. The control unit 210 is, for example, a Central Processing Unit (CPU) (not shown). The control unit 210 realizes various functions related to the information processing device 200 by reading predetermined programs stored in the storage unit 220. That is, information processing by software stored in the storage unit 220 is concretely realized by the control unit 210, which is an example of hardware, and can be executed as each functional unit included in the control unit 210. These will be explained further in Section 2. Note that the control unit 210 is not limited to being a single unit, and may be implemented with multiple control units 210 for each function, or a combination thereof.

[0024] The storage unit 220 stores various information necessary for information processing by the information processing device 200. This can be implemented, for example, as a storage device such as a solid-state drive (SSD) that stores various programs related to the information processing device 200 executed by the control unit 210, or as memory such as random access memory (RAM) that stores temporarily necessary information (arguments, arrays, etc.) related to program calculations. A combination of these may also be used.

[0025] The communication unit 250 preferably uses wired communication methods such as USB, IEEE 1394, Thunderbolt®, and wired LAN network communication, but may also include wireless LAN network communication, mobile communication such as 5G / LTE / 3G, and Bluetooth® communication as needed. In other words, it is more preferable to implement it as a collection of these multiple communication methods. That is, the information processing device 200 communicates various information with the terminal 300 via the network through the communication unit 250.

[0026] 1-3. Terminal 300 Figure 3 is a block diagram showing the hardware configuration of terminal 300. Terminal 300 includes a control unit 310, a storage unit 320, a display unit 330, an input unit 340, and a communication unit 350, and these components are electrically connected within terminal 300 via a communication bus 360. The explanation of the control unit 310, storage unit 320, and communication unit 350 is substantially the same as the explanation of the control unit 210, storage unit 220, and communication unit 250 in the information processing device 200, so it is omitted here.

[0027] The display unit 330 may be included in the casing of the terminal 300 or it may be an external component. The display unit 330 displays a graphical user interface (GUI) screen that can be operated by the user. This is preferably implemented by using a display device such as a CRT display, liquid crystal display, organic EL display, and plasma display, depending on the type of terminal 300. In the following description, the display unit 330 will be described as being included in the casing of the terminal 300.

[0028] The input unit 340 may be included in the casing of the terminal 300 or it may be an external component. For example, the input unit 340 may be integrated with the display unit 330 and implemented as a touch panel. If it is a touch panel, the user can input tap operations, swipe operations, etc. Of course, instead of a touch panel, a switch button, mouse, QWERTY keyboard, etc., may be used. In other words, the input unit 340 receives operation input made by the user. This input is transmitted as a command signal to the control unit 310 via the communication bus 360. The control unit 310 can then perform predetermined controls and calculations as needed.

[0029] 2. Functional Configuration Section 2 describes the functional configuration of this embodiment. As described above, the information processing by the software stored in the memory unit 220 is specifically realized by the control unit 210, which is an example of hardware, and can be executed as each functional unit included in the control unit 210.

[0030] FIG. 4 is a block diagram showing functions implemented by the information processing apparatus 200 (control unit 210). As described above, the information processing apparatus 200 (information processing system 100) includes a control unit 210. Specifically, the information processing apparatus 200 (control unit 210) is configured to execute each step in the information processing method of the present embodiment. The information processing apparatus 200 (control unit 210) includes an acquisition unit 211, a generation unit 212, a conversion unit 213, an output unit 214, and a creation unit 215 corresponding to each step in the information processing method of the present embodiment.

[0031] The acquisition unit 211 is configured to acquire various information. The acquisition unit 211 is configured to execute an acquisition step. For example, the acquisition unit 211 acquires a sensor point cloud layer defined as a layer of point clouds representing the arrangement of a plurality of sensors, a target point cloud layer defined as a layer of point clouds representing a region to be sensed, and measurement data measured by each sensor. Here, a physical distance is defined to exist between the sensor point cloud layer and the target point cloud layer. The measurement data indicates a raw signal measured and acquired by the sensor. The measurement data may be a time series having a time length or may not have a time length. Further, the measurement data may have dimensions of feature amounts other than time or may have only one-dimensional feature amounts.

[0032] The generation unit 212 is configured to generate various information. The generation unit 212 is configured to execute a generation step. For example, the generation unit 212 generates one or more intermediate point cloud layers defined as layers of point clouds located between the sensor point cloud layer and the target point cloud layer.

[0033] The conversion unit 213 is configured to convert various information. The conversion unit 213 is configured to execute a conversion step. For example, the conversion unit 213 propagates each measurement data as a feature amount of each point included in the intermediate point cloud layer step by step from the sensor point cloud layer toward the target point cloud layer via the intermediate point cloud layer, and converts each propagated feature amount into an attribute value of each point in the target point cloud layer.

[0034] The output unit 214 is configured to output various types of information. The output unit 214 is configured to execute an output step. For example, the output unit 214 outputs the attribute values of each of the converted points. The attribute value is the final desired data for confirming the phenomenon occurring in the sensing target, and is, for example, the membrane potential of the heart. The attribute value is not particularly limited as long as it indicates some physical quantity. The attribute value may be a time series having a time length, or may not have a time length. Further, the attribute value may have dimensions of feature amounts other than time, or may have only one-dimensional feature amounts.

[0035] The creation unit 215 is configured to create various types of information. The creation unit 215 is configured to execute a creation step. For example, the creation unit 215 creates a target data set that is a data set of the attribute values of the sensing target, creates a sensor data set that is a data set of the coordinates where a plurality of sensors are arranged, and further creates a measurement data set that is a data set of the measurement data measured at different heights by a plurality of sensors.

[0036] 3. Information Processing Method In Section 3, the flow of the information processing method of the above-described information processing apparatus 200 will be described. This information processing method includes an acquisition step, a generation step, a conversion step, and an output step.

[0037] FIGS. 5 and 6 are activity diagrams showing the flow of the information processing method executed by the information processing apparatus 200. Hereinafter, the description will be made along with each activity of these activity diagrams.

[0038] First, the control unit 310 in terminal 300 prepares the sensor point cloud layer, the target point cloud layer, and the measurement data (activity A110). The sensor point cloud layer, the target point cloud layer, and the measurement data may be created, for example, by a simulator capable of reproducing the physical signal measurement process. In activity A110, for example, the following three stages of information processing are performed: (1) The control unit 310 reads the simulator from the storage unit 320. (2) The control unit 310 operates the simulator to create the sensor point cloud layer, the target point cloud layer, and the measurement data. (3) The control unit 310 stores the sensor point cloud layer, the target point cloud layer, and the measurement data in the storage unit 320.

[0039] Next, the control unit 310 in terminal 300 transmits the sensor point cloud layer, target point cloud layer, and measurement data to the information processing device 200 (activity A120). In activity A120, for example, the following two stages of information processing are performed: (1) The control unit 310 reads the sensor point cloud layer, target point cloud layer, and measurement data from the storage unit 320. (2) The control unit 310 transmits the sensor point cloud layer, target point cloud layer, and measurement data to the information processing device 200 via the communication unit 350.

[0040] Next, the control unit 210 in the information processing device 200 receives the sensor point cloud layer, the target point cloud layer, and measurement data from the information processing device 200 (activity A130). To put this in terms of steps, in the acquisition step, the sensor point cloud layer, defined as a layer of point clouds representing the arrangement of multiple sensors, the target point cloud layer, defined as a layer of point clouds representing the area to be sensed, and the measurement data measured by each sensor are acquired. Here, a physical distance is defined to exist between the sensor point cloud layer and the target point cloud layer. Here, the physical distance between the sensor point cloud layer and the target point cloud layer is defined as a concept that includes 0. That is, it includes the case where the physical distance is 0 when the sensor is placed in contact with the surface of the sensing target, and the case where the physical distance is a value greater than 0 when the sensor is placed at a distance from the surface of the sensing target, in a so-called non-contact state.

[0041] In Activity A130, for example, the following two-stage information processing is performed: (1) The communication unit 250 receives the sensor point cloud layer, target point cloud layer, and measurement data from the information processing device 200. (2) The control unit 210 stores the sensor point cloud layer, target point cloud layer, and measurement data in the storage unit 220.

[0042] Figure 7 shows an example of a sensor point cloud layer, a target point cloud layer, and measurement data. Figure 7(A) shows the positional relationship between the sensor point cloud layer 410 and the target point cloud layer 420. The target point cloud layer 420 is defined as a 15 × 15 mm tissue surface for sensing. The sensor point cloud layer 410 is defined as an arrangement of 16 sensors positioned at a height of 15 mm from the tissue surface. Figure 7(B) shows the measurement data 430 measured by the 16 sensors in the sensor point cloud layer 410.

[0043] Returning to the explanation of Figure 5, the control unit 210 in the information processing device 200 extracts the feature quantities of each point in the sensor point cloud layer (activity A140). In other words, in the conversion step, a convolutional neural network model is used to extract the measurement data of each point in the sensor point cloud layer and the feature quantities of each point in the intermediate point cloud layer. In the detailed explanation of activity A140, an example of processing using a convolutional neural network model will be described.

[0044] Here, using the activity diagram in Figure 6, the details of Activity A140 will be explained. The control unit 210 in the information processing device 200 arbitrarily selects a first point from the point cloud in the sensor point cloud layer (Activity A141). To put this in terms of steps, in the conversion step, a first point is selected from the point cloud in the sensor point cloud layer or the intermediate point cloud layer. In Activity A141, for example, the following three stages of information processing are executed: (1) The control unit 210 reads the sensor point cloud layer from the storage unit 220. (2) The control unit 210 performs a selection process and selects a first point. (3) The control unit 210 stores information that a first point has been selected (hereinafter also referred to as "first selection information") in the storage unit 220.

[0045] Next, the control unit 210 in the information processing device 200 calculates the relative coordinates of the first point based on the coordinates of the first point and the coordinates of one or more second points adjacent to the first point (Activity A142). In other words, in the conversion step, the relative coordinates of the first point are calculated based on the coordinates of the first point and the coordinates of one or more second points adjacent to the first point. In Activity A142, for example, the following three stages of information processing are performed: (1) The control unit 210 reads the first selection information from the storage unit 220. (2) The control unit 210 performs a selection process and selects a second point adjacent to the first point. (3) The control unit 210 performs a calculation process using the first point and the second point to calculate the relative coordinates of the first point. (4) The control unit 210 stores the relative coordinates of the first point in the storage unit 220.

[0046] Next, the control unit 210 in the information processing device 200 calculates a weight coefficient (hereinafter also simply referred to as "weight coefficient") corresponding to the first point using MLP (Multi-layer Perception) calculations with the relative coordinates of the first point (Activity A143). To put this in terms of steps, in the conversion step, the weight coefficient corresponding to the first point is calculated based on the relative coordinates of the first point and the MLP (corresponding to the "mathematical model" in the claims). Here, the mathematical model is a model that shows the relationship between the relative coordinates of a certain point and the weight coefficient corresponding to that point. Taking the above MLP as an example, it is a model described so that when the relative coordinates of a certain point are input, the weight coefficient corresponding to that point is output. In Activity A143, for example, the following three stages of information processing are executed: (1) The control unit 210 reads the relative coordinates of the first point and the MLP from the storage unit 220. (2) The control unit 210 calculates the relative coordinates of the first point using MLP and calculates the weight coefficients. (3) The control unit 210 stores the weight coefficients in the storage unit 220.

[0047] Next, the control unit 210 in the information processing device 200 performs a sum-of-products operation on the calculated weight coefficient and the measurement data of the second point to extract the feature quantity of the first point (Activity A144). In other words, in the conversion step, the feature quantity of the first point is extracted based on the weight coefficient corresponding to the first point and the measurement data or feature quantity of the second point. In Activity A144, for example, the following three stages of information processing are executed: (1) The control unit 210 reads the weight coefficient and the measurement data of the second point from the storage unit 220. (2) The control unit 210 performs a sum-of-products operation on the weight coefficient and the measurement data of the second point to extract the feature quantity of the first point. (3) The control unit 210 stores the feature quantity of the first point in the storage unit 220.

[0048] Next, the control unit 210 in the information processing device 200 determines whether or not it has extracted feature quantities for all points within the same layer (activity A145). If there are points from which feature quantities have not been extracted, the control unit 210 proceeds to the processing of activity A141, executes the processing of activities A141 to A144, and then executes the processing of activity A145 again (NO for activity A145). On the other hand, if feature quantities for all points have been extracted, the control unit 210 proceeds to the processing of activity A150 (YES for activity A145).

[0049] Thus, according to Activity A140 (Activities A141 to A145), it is possible to estimate the spatiotemporal distribution of physical quantities in real space from multiple measurement data.

[0050] Figure 8 is a conceptual diagram corresponding to activity A140. First, in activity A141, the first point 411 is selected (see Figure 8(A)). Next, in activity A142, the second points 412, 413, 414, and 415, which are adjacent to the first point 411, are selected (see Figure 8(B)). In activity A142, the second points 412, 413, 414, and 415 are grouped together to form the second point group 416 (see Figure 8(B)). In activity A142, the relative coordinates of the first point 411 are calculated using the first point 411 and the second point group 416. Next, in activity A143, the weight coefficient k corresponding to the first point 411 is calculated using MLP with the relative coordinates of the first point 411. Next, in activity A144, the weight coefficient k and the measurement data corresponding to each second point shown in Figure 8(C) are used in a sum-of-products operation to extract the feature quantities of the first point 411. Then, the processing of activities A141 to A144 is repeated until the feature quantities of all points within the same layer are extracted (activity A145).

[0051] Returning to the explanation of Figure 5, the control unit 210 in the information processing device 200 generates an intermediate point cloud layer closer to the target point cloud layer than the previous layer (sensor point cloud layer or intermediate point cloud layer) (Activity A150). In other words, the generation step generates one or more intermediate point cloud layers defined as layers of point clouds located between the sensor point cloud layer and the target point cloud layer. Here, the control unit 210 interpolates new points to increase the density of the point cloud contained in the previous layer (Activity A150). In other words, the generation step generates intermediate point cloud layers while increasing the density of the point cloud as you move from the sensor point cloud layer towards the target point cloud layer. That is, if the number of points in the target point cloud layer is greater than the number of points in the previous layer, the number of points in the previous layer and the number of points in the target point cloud layer are compared, and the number of points in the intermediate point cloud layer is gradually increased so that the number of points when the target point cloud layer is reached matches. To put this in terms of steps, in the generation step, when generating a new intermediate point cloud layer, new points are interpolated based on the position information of each point in the previous layer and the position information of each point in the target point cloud layer.

[0052] In Activity A150, for example, the following three stages of information processing are performed: (1) The control unit 210 reads the position information of each point in the previous layer, the position information of each point in the target point cloud layer, and a pre-set augmentation algorithm from the storage unit 220. (2) The control unit 210 executes the augmentation algorithm and interpolates new points using the position information of each point in the previous layer and the position information of each point in the target point cloud layer. (3) The control unit 210 stores the newly interpolated points in the storage unit 220. According to Activity A150, even if there is a discrepancy between the number of points (sensors) in the sensor point cloud layer and the number of points (measurement points) in the target point cloud layer, the accuracy of the measurement can be ensured by applying an intermediate point cloud layer as a buffer. Furthermore, it can be adjusted to match the final number of points (measurement points) in the target point cloud layer.

[0053] Next, the control unit 210 in the information processing device 200 propagates the feature quantities of each point in the previous layer to the newly generated intermediate point cloud layer (Activity A152). In other words, in the conversion step, the feature quantities of each point in the previous layer are propagated to the newly generated intermediate point cloud layer. That is, in the conversion step, the measurement data is propagated step by step from the sensor point cloud layer to the target point cloud layer, via the intermediate point cloud layer, as the feature quantities of each point in the intermediate point cloud layer. In Activity A152, for example, the following three stages of information processing are executed: (1) The control unit 210 reads the feature quantities of each point in the previous layer from the storage unit 220. (2) The control unit 210 propagates the corresponding feature quantities to each point in the intermediate point cloud layer. (3) The control unit 210 stores the intermediate point cloud layer with the propagated feature quantities in the storage unit 220. According to Activity A152, the accuracy of the feature quantities of each point in each layer can be ensured by propagating the extracted feature quantities.

[0054] Figure 9 is a conceptual diagram corresponding to activities A150 and A152. Here, the reference layer is the intermediate point cloud layer 441. First, in activity A150, a new point 451 is interpolated to increase the number of points in the sensor point cloud layer 410 (increase the density of the point cloud). Here, the intermediate point cloud layer 441, which includes the interpolated new point 451, is generated. Next, in activity A152, the feature quantities of each point in the previous layer, the sensor point cloud layer 410, are propagated to the position corresponding to the intermediate point cloud layer 441. Here, the feature quantities of each point are extracted by performing an inverse convolution operation using, for example, a weight coefficient k calculated for several points in the sensor point cloud layer 410. The intermediate point cloud layer generation process is performed in such a way as to reduce the physical distance from the sensor point cloud layer 410 to the target point cloud layer 420, as in the intermediate point cloud layer 442 with the interpolated new point 452, and the intermediate point cloud layer 443 with the interpolated new point 453.

[0055] Returning to the explanation of Figure 5, the control unit 210 in the information processing device 200 then determines whether the generated intermediate point cloud layer has been generated at the location of the target point cloud layer (Activity A160). If the generated intermediate point cloud layer has not yet been generated at the location of the target point cloud layer, the control unit 210 proceeds to the processing of Activity A140, executes the processing from Activities A140 to A152, and then executes the determination processing of Activity A160 again (Activity A160 NO). On the other hand, if the generated intermediate point cloud layer has been generated at the location of the target point cloud layer, the control unit 210 proceeds to the processing of Activity A170 (Activity A160 YES).

[0056] Next, the control unit 210 in the information processing device 200 converts the feature quantities of each point included in the last generated intermediate point cloud layer into attribute values ​​(activity A170). In other words, in the conversion step, each feature quantity propagated to the target point cloud layer is converted into an attribute value for each point in the target point cloud layer. In activity A170, for example, the following four stages of information processing are performed: (1) The control unit 210 reads the feature quantities of each point included in the last generated intermediate point cloud layer from the storage unit 220. (2) The control unit 210 reads a predetermined module or model from the storage unit 220. (3) The control unit 210 performs a conversion process using the module or model and converts each feature quantity into an attribute value for each point in the target point cloud layer. (4) The control unit 210 stores the attribute values ​​of each point in the storage unit 220.

[0057] Figure 10 shows the state in which the feature quantities of each point included in the intermediate point cloud layer have been converted into attribute values. In Figure 9, when the intermediate point cloud layer 443 is generated at the position of the target point cloud layer 420, the feature quantities of each point included in the intermediate point cloud layer 443 are converted into attribute values ​​421. Here, it is assumed that the number of points included in the intermediate point cloud layer 443 is the same as the number of points included in the target point cloud layer 420. In other words, if activity A170 is translated into steps, in the generation step, when the position of the intermediate point cloud layer reaches the position of the target point cloud layer, the intermediate point cloud layer is generated so that the number of points included in the intermediate point cloud layer matches the number of points included in the target point cloud layer. With this configuration, even with a sparse sensor arrangement, it is possible to obtain a number of attribute values ​​corresponding to the number of points (measurement points) in the target point cloud layer.

[0058] Returning to the explanation of Figure 5, the control unit 210 in the information processing device 200 generates display information (Activity A180) for displaying the attribute values ​​of each point in the target point cloud layer on the display unit 330 of the terminal 300. In Activity A180, for example, the following three stages of information processing are performed: (1) The control unit 210 reads the attribute values ​​of each point in the target point cloud layer from the storage unit 220. (2) The control unit 210 executes a generation process to generate display information that visualizes the attribute values. (3) The control unit 210 stores the display information in the storage unit 220.

[0059] Next, the control unit 210 in the information processing device 200 transmits the attribute value display information to the terminal 300 (activity A190). In other words, in the output step, the converted attribute value is output. In activity A190, for example, the following two stages of information processing are performed: (1) The control unit 210 reads the attribute value display information from the storage unit 220. (2) The control unit 210 transmits the attribute value display information to the terminal 300 via the communication unit 250.

[0060] Next, the control unit 310 in terminal 300 receives the attribute value display information from the information processing device 200 (Activity A200). In Activity A200, for example, the following two stages of information processing are performed: (1) The communication unit 350 receives the attribute value display information from the information processing device 200. (2) The control unit 310 stores the attribute value display information in the storage unit 320.

[0061] Next, the control unit 310 in terminal 300 displays the attribute value information on the display unit 330 (activity A210). In activity A210, for example, the following three stages of information processing are performed: (1) The control unit 310 reads the attribute value information from the storage unit 320. (2) The control unit 310 transmits the attribute value information to the display unit 330 via the communication bus 360. (3) The display unit 330 displays an image based on the attribute value information.

[0062] According to the information processing method described above, in measurements requiring image reconstruction, it is possible to ensure measurement accuracy while solving at least one of the following (1) and (2): (1) Improving the degree of freedom of the physical distance between the sensor and the sensing object compared to conventional methods. (2) Arranging sensors more sparsely than conventional methods. Furthermore, due to its simple configuration, the saved resources can be used for other core functions.

[0063] 4. Example 1 Section 4 will describe Example 1 of this embodiment.

[0064] 4-1. Method for Generating a Trained Model Figure 11 is an activity diagram showing the flow of the method for generating a trained model, which is executed by terminal 300. Figure 12 is a diagram showing an example of a dataset created in the method for generating a trained model. Hereinafter, each activity in the activity diagram of Figure 11 will be explained with reference to Figure 12. In this embodiment, the method for generating a trained model is configured to use a computer to execute the creation step and the generation step. Here, the process of generating a trained model using terminal 300 as an example of a computer will be described.

[0065] First, the control unit 310 in terminal 300 used the atrial myocyte model by Courtemanche et al. (hereinafter also referred to as the "atrial myocyte model"), which was applied to locally proliferated fibroblasts, to create a target dataset 510 (see Figure 12(A)), which is a dataset of the membrane potential of the atrial myocyte model (Activity A310). This atrial myocyte model simulates a human atrial tissue model of heart failure. To put this in terms of steps, the creation step created a target dataset, which is a dataset of attribute values ​​of the sensing target. In Activity A310, for example, the following four stages of information processing were performed: (1) The control unit 310 read the simulator from the storage unit 320. (2) The control unit 310 read the atrial myocyte model from the storage unit 320. (3) The control unit 310 applied the atrial myocyte model to the simulator and simulated the changes in membrane potential at 1024 points on the tissue surface. (4) The control unit 310 stored the changes in each membrane potential as a target dataset 510 in the storage unit 320.

[0066] Next, the control unit 310 in terminal 300 created a sensor dataset 520 (see Figure 12(B)) which is a dataset of coordinates of 16 sensors placed at a height of 0.5 mm from the surface of the atrial muscle cell model (hereinafter also referred to as "0.5 mm coordinates"), coordinates of 16 sensors placed at a height of 5 mm (hereinafter also referred to as "5 mm coordinates"), and coordinates of 16 sensors placed at a height of 15 mm (hereinafter also referred to as "15 mm coordinates") (Activity A320). To put this in terms of steps, the creation step created a sensor dataset, which is a dataset of coordinates where multiple sensors are placed. In Activity A320, for example, the following two stages of information processing were performed: (1) The input unit 340 received input of 0.5 mm coordinates, 5 mm coordinates, and 15 mm coordinates. (2) The control unit 310 stored the 0.5 mm coordinates, 5 mm coordinates, and 15 mm coordinates in the storage unit 320 as sensor dataset 520, respectively.

[0067] Next, the control unit 310 in terminal 300 created a measurement dataset 530 (see Figure 12(C)) which is a dataset of measurement data measured by 16 sensors placed at a height of 0.5 mm from the surface of the atrial myocyte model (hereinafter also referred to as "0.5 mm measurement data"), measurement data measured by 16 sensors placed at a height of 5 mm (hereinafter also referred to as "5 mm measurement data"), and measurement data measured by 16 sensors placed at a height of 15 mm (hereinafter also referred to as "15 mm measurement data") (Activity A330). In other words, in the creation step, a measurement dataset was created which is a dataset of measurement data measured at different heights by multiple sensors. In Activity A330, for example, the following four stages of information processing were performed: (1) The control unit 310 read the simulator, the target dataset 510, and the sensor dataset 520 from the storage unit 320. (2) The control unit 310 applied the target dataset 510 and the sensor dataset 520 to the simulator and simulated the measurement data from 16 sensors on the tissue surface under different height conditions. (4) The control unit 310 stored each measurement data as a measurement dataset 530 in the storage unit 320.

[0068] Next, the control unit 310 in terminal 300 generated the trained model of Embodiment 1 by performing deep learning using the target dataset 510, sensor dataset 520, and measurement dataset 530 (Activity A340). In other words, in the generation step, a trained model was generated based on the target dataset, sensor dataset, and measurement dataset. Here, this trained model is a model that outputs attribute values ​​for the input of coordinates and measurement data of each sensor. In Activity A340, for example, the following four stages of information processing were performed: (1) The control unit 310 read the target dataset 510, sensor dataset 520, and measurement dataset 530 from the storage unit 320. (2) The control unit 310 read the neural network. (3) The control unit 310 fed the target dataset 510, sensor dataset 520, and measurement dataset 530 into the neural network and generated a trained model. (4) The control unit 310 stored the trained model in the storage unit 320.

[0069] 4-2. Validation of the Trained Model Figure 13 shows the accuracy of the prediction results by the trained model generated in Section 4-1. As shown in Figure 13, both the mean value and standard deviation (SD) of the mean absolute error (MAE) increase as the sensor height increases, but even when the sensor height is 15 mm, the mean error remains below 2.5 mV.

[0070] Figure 14 shows the results of membrane potential mapping using the trained model generated in Section 4-1. As shown in Figure 14(A), the measurement data becomes blurry as the sensor height increases. Figure 14(B) shows the predicted membrane potential at the dashed-line-enclosed area of ​​the measurement data in Figure 14(A). As shown in Figure 14(B), it was found that even from blurry measurement data at a height of 15 mm, the trained model can accurately predict the complex dynamics of helical wave collisions caused by wave breaking.

[0071] Example 1 in Section 4 demonstrated the possibility of deep learning-based membrane potential mapping from non-contact sensor signals. Furthermore, this method for generating the trained model provides a technology that utilizes the information processing method of this embodiment.

[0072] 5. Example 2 Section 5 describes Example 2 of this embodiment.

[0073] Figure 15 shows the process of generating intermediate point cloud layers using the trained model of Example 2. In Figure 15, a sensor point cloud layer 610 and a target point cloud layer 620 were prepared, and three intermediate point cloud layers (intermediate point cloud layer 641, intermediate point cloud layer 642, and intermediate point cloud layer 643) were generated between the sensor point cloud layer 610 and the target point cloud layer 620. On the other hand, in the comparative example shown in Figure 17, described later, each measurement data contained in the sensor point cloud layer 610 was directly propagated to the target point cloud layer 620 as a feature without passing through intermediate point cloud layers.

[0074] Figure 16 shows an overview of the training dataset used to generate the trained model in Example 2. The prepared training dataset was created by randomly applying the height, inclination, and rotation angle of each point in the sensor point cloud layer 610 to the target point cloud layer 620. Training dataset 1 had a height of 1.8 to 4.5 mm, an inclination of 0 to 5°, and a rotation angle of 0 to 360°. Training dataset 2 had a height of 1.8 to 22.5 mm, an inclination of 0 to 5°, and a rotation angle of 0 to 360°. Training dataset 3 had a height of 1.8 to 45 mm, an inclination of 0 to 5°, and a rotation angle of 0 to 360°. Each training dataset contained 11,060 training data points and 1,382 evaluation data points.

[0075] Figure 17 shows a comparison of the case where features are propagated via an intermediate point cloud layer and the case where features are propagated directly without going through the intermediate point cloud layer, using the trained model of Example 2. In both training datasets, the average mean absolute error (MAE) was smaller when features were propagated via the intermediate point cloud layer compared to when they were not. In particular, a significant difference in MAE was observed when the upper limit of height was large (training datasets 2 and 3).

[0076] According to Example 2 in Section 5, it was demonstrated that by gradually propagating feature quantities via an intermediate point cloud layer, it is possible to reduce the influence of attenuation of the measurement signal depending on the distance between the sensor placement and the area to be sensed.

[0077] 6. Section 6 of the second embodiment describes the second embodiment. In the second embodiment, explanations of parts that overlap with the first embodiment will be omitted as appropriate.

[0078] 6-1. Background diagram 18 is a diagram illustrating the time shift of ultrasonic signals. In the first embodiment, electrical signals were used as an example of measurement data measured by each sensor. However, measurement data includes various signals other than electrical signals, and signals that have characteristics that require time to propagate depending on the physical distance, such as ultrasonic signals, need to be treated differently from electrical signals.

[0079] Ultrasound-based diagnostic imaging is widely used in many medical settings because it is relatively inexpensive and minimally invasive. In recent years, flexible ultrasound probes that can be attached to the body have attracted attention for the purpose of routine monitoring within the body.

[0080] However, densely arranging sensors on a flexible ultrasonic probe leads to complex wiring and reduced flexibility. Therefore, sparsely arranging the sensors is conceivable, but this is difficult when using conventional methods for reconstructing ultrasonic images (Delay & Sum method) because artifacts tend to occur in the image.

[0081] Therefore, we considered an algorithm that applies the concept of signal time shifting, as shown below.

[0082] For example, as shown in Figure 18(A), when each sensor element (sensor 701, sensor 702, sensor 703, sensor 704, sensor 705, sensor 706, sensor 707, and sensor 708) is arranged in the X-axis direction, when ultrasonic waves are generated from the sound source 760, each sensor element will receive the ultrasonic signal at a different time because they are at different distances from the sound source 760. Furthermore, since the reception time of the ultrasonic signal at each sensor element includes characteristic quantities of the sound source 760, it was considered necessary to time-shift the received signal in order to extract these characteristic quantities.

[0083] Therefore, in this embodiment, we decided to use the Fourier transform to represent the time shift of the signal. That is, since the Fourier transform is a transformation formula that focuses on the relationship between the time domain and the frequency domain, we thought it could be applied to ultrasonic signals where a time delay occurs depending on the distance. In this embodiment, the sensor is also called a "sensor element" or simply an "element".

[0084] Here, the ultrasonic signal is denoted by equation (1) and explained below.

[0085] ... (1)

[0086] First, when the ultrasonic signal in equation (1) is Fourier transformed, the ultrasonic signal is represented as a complex number array in frequency space (see equation (2)).

[0087] ... (2)

[0088] Next, the time to shift the complex number array in equation (2) is t. ０ Multiplying by the complex number in equation (3) results in a weighted array of complex numbers (see equation (4)).

[0089] ... (3)

[0090] ... (4)

[0091] Next, when we perform an inverse Fourier transform on the complex number array in equation (4), we get time t ０ A time-shifted signal is obtained.

[0092] ... (5)

[0093] Here, an ultrasonic signal is generated from the sound source 760 at a certain time t, and the signals acquired by all sensors are defined as signal 731 (see Figure 18(B)). Then, for all the signals acquired by each sensor element, time t ０ By shifting the time by that amount, a signal 732 was obtained in which the entire signal 731 was translated parallel to the time axis (see Figure 18(C)).

[0094] Furthermore, by adjusting the amount of time shift for each sensor, the time axis of the signals can be aligned. In Figure 18(D), the amount of time shift is set to 0 for the signal acquired by the sensor closest to the sound source 760 (for example, sensor 705 in Figure 18(A)), and the amount of time shift is set to increase according to the distance from sensor 705 (for example, the amount of time shift is set to increase in the order of sensor 704 (sensor 706), sensor 703 (sensor 707), sensor 702 (sensor 708), and sensor 701), thereby obtaining a signal 733 with aligned time axes.

[0095] The proposed method, which applies each of the above processes, is described below.

[0096] 6-2. Proposed Method Figure 19 is a diagram illustrating the process performed when extracting feature quantities from each point. First, the first point 711 is selected (see Figure 19(A)). Next, the second points 712, 713, 714, and 715, which are adjacent to the first point 711, are selected (see Figure 19(B)). Then, the first point 711, the second point 712, the second point 713, the second point 714, and the second point 715 are grouped together to form the second point group 716 (see Figure 19(B)).

[0097] Next, the relative coordinates of the first point 711 are calculated using the first point 711 and the second point group 716. Then, using the relative coordinates of the first point 711, the weight coefficient k corresponding to the first point 711 is calculated using MLP for the second points 712, 713, 714, and 715. ｉ The weight coefficient k is calculated. ｉIt includes a coefficient related to the amplitude of the measurement data and a coefficient related to time. Here, the coefficient related to the amplitude is the amplitude magnification factor A shown in Equation (6). ｉ is defined as, and the coefficient related to time is the time shift amount T shown in Equation (7). ｉ is defined as.

[0098] ・・・(6)

[0099] ・・・(7)

[0100] Here, (dx ｉ , dy ｉ , dz ｉ ) in Equation (6) and Equation (7) represent the relative coordinates between the second point i and the first point 711.

[0101] Subsequently, the amplitude of the signal is multiplied by A ｉ times, and a complex number array k ｉ that is time-shifted by T ｉ is generated (see Equation (8)). This complex number array k ｉ is synonymous with the weight coefficient k ｉ .

[0102] ・・・(8)

[0103] Subsequently, using the complex number array k ｉ , a sum-product operation is performed on the signal of the second point (see Equation (9)).

[0104] ・・・(9)

[0105] Through the above processing, convolution can be performed while adjusting the time axis of the signal.

[0106] FIG. 20 is a diagram for explaining the processing executed when interpolating new points. Here, the reference layer is the intermediate point group layer 741. First, new points 751 are interpolated so as to increase the number of point groups (increase the density of the point groups) included in the sensor point group layer 710. Here, an intermediate point group layer 741 including the interpolated new points 751 is generated. Subsequently, the feature amounts of each point included in the sensor point group layer 710, which is the previous layer, are propagated to the corresponding positions in the intermediate point group layer 741.

[0107] Here, taking the new point 751 as an example, the distance r from the new point 751 to the neighboring point (the neighboring point corresponding to the new point 751) included in the sensor point cloud layer 710 is... ｌ Amplitude multiplier B using the reciprocal of ｉ and distance r ｌ Time shift amount T obtained by dividing by the speed of sound c. ｉ The following was calculated.

[0108] ... (10)

[0109] ... (11)

[0110] Next, the amplitude of the signal is B ｉ Double the signal to T ｉ A complex number array k that is time-shifted by only that amount. ｉ This generates the complex number array k (see equation (12)). ｉ The weight coefficient k is ｉ It is synonymous with [the above].

[0111] ... (12)

[0112] Next, the complex number array k ｉ Using this, a sum-of-products operation is performed on the signal at the new point 751 (see equation (13)).

[0113] ... (13)

[0114] As a result of the above processing, if the scattering point, which is the sound source of the reflected wave, is nearby, multiple waves will reinforce each other at the new point 751 at the same time. Therefore, after processing up to equation (13), by using an activation function to remove the weakly overlapping signal portion, it is possible to make it so that a large amplitude is observed only at positions close to the scattering point.

[0115] In this way, the intermediate point cloud layer generation process is performed in such a way as to reduce the physical distance from the sensor point cloud layer 710 to the target point cloud layer 720, as shown by the intermediate point cloud layer 742 being created by interpolating the new point 752, and the intermediate point cloud layer 743 being created by interpolating the new point 753.

[0116] 6-3. Outline of the Experimental Example Figure 21 is a diagram illustrating the outline of the experimental example in this embodiment. To verify the principle of the proposed method described in Section 6-2, a simulation experiment was conducted under simple conditions. As shown in Figure 21(A), several ultrasonic sound sources (sound sources 761, 762, and 763) were placed in the imaging area 730 instead of scattering points, and ultrasonic waves were generated simultaneously. Then, ultrasonic waves were received by a plurality of sensors 770 unevenly arranged in the imaging area 730, and this ultrasonic signal was used as input. As shown in Figure 21(A), the entire imaging area 730 was considered as a point cloud of virtual sensors 780, and the virtual signals estimated for the entire point cloud were used as output.

[0117] In Figure 21(B), the virtual signal was estimated by performing a Fourier transform on the signal received by sensor 770, executing deep learning in the frequency domain, and then performing an inverse Fourier transform on the complex number array to return to the real signal. In Figure 21(C), the estimated virtual signal was converted into image 790 to identify the locations of the ultrasonic sound sources (locations 791, 792, and 793).

[0118] Figure 22 is a diagram illustrating a method for converting estimated virtual signals into images. Figure 22(A) shows an enlarged view of the sound source 761 and the virtual sensors 780 arranged around the sound source 761. First, when ultrasonic waves (spherical waves) are generated from the sound source 761, a large amplitude is observed at t=0 in the virtual sensor 781, which is located at the same position as the sound source 761. Then, the ultrasonic waves generated from the sound source 761, as signals with attenuated amplitude, propagate later than t=0 to the virtual sensors 782, 783, and 784, which are located spaced apart from the sound source 761.

[0119] In this experiment, since pulses are generated simultaneously from all sound sources at time t=0, if the signals are ideally estimated, points with large amplitudes around t=0 can be expected to be the sound sources. Therefore, in this experiment, the maximum amplitude from t=0 to a sampling interval of 10 was converted into the brightness of an image as shown in Figure 22(B). Then, the mean squared error (MSE) of the brightness of each pixel in the converted image was used to compare the image quality of the conventional method with that of the proposed method.

[0120] Figure 23 shows an example of sensor placement in the training dataset. In this experiment, training datasets with modified sound source and sensor placements were used. Specifically, for the sound source placement, 1 to 4 sound sources were randomly placed within the imaging area 730. For the sensor placement, 256 sensors arranged as a dense 16x16 rectangular array were thinned using the farthest point sampling method to obtain three placement patterns: 16 sensors (see Figure 23(A)), 32 sensors (see Figure 23(B)), and 128 sensors (see Figure 23(C)). Using these placement patterns, ultrasonic reception simulations were performed to create 2000 training datasets.

[0121] Figure 24 shows the conditions for an experimental example in this embodiment. In this experimental example, in order to simplify and visualize the learning process, a method was adopted to directly estimate the signal in the target point cloud layer 720 from the ultrasonic signal received by the sensor point cloud layer 710, without passing through an intermediate point cloud layer. Figure 24(A) shows an example in which 16 sensors 770 are arranged in the sensor point cloud layer 710. Here, the ultrasonic signals from each sensor 770 in the sensor point cloud layer 710 were propagated to the target point cloud layer 720 using the proposed method of this embodiment. Subsequently, spatial convolution was performed in the target point cloud layer 720 to obtain virtual signals for 256 elements. Figure 24(B) shows the state in which virtual signals have been obtained in the target point cloud layer 720.

[0122] Figure 24(C) shows the changes in the number of points, the number of frequency components, and the dimension of the feature vector when a signal is propagated from the sensor point cloud layer 710 to the target point cloud layer 720 using the proposed method of this embodiment. First, in the sensor point cloud layer 710, the number of points was set to 16, the number of frequency components to 256, and the dimension of the feature vector to 1. Next, a process of interpolating new points was performed to increase the number of points to 256. Subsequently, spatial convolution was performed to increase the dimension of the feature vector to 8, and then finally to decrease the dimension of the feature vector to 1. In this way, a virtual signal was obtained that was a sparse sensor arrangement in the sensor point cloud layer 710 and a dense sensor arrangement in the target point cloud layer 720.

[0123] Table 1 shows the parameters used in the simulation, and Table 2 shows the parameters used in training.

[0124]

[0125] In Table 1, the sensor point cloud indicates the number of sensors 770 in the sensor point cloud layer 710, and the target point cloud indicates the number of virtual sensors 780 in the target point cloud layer 720.

[0126]

[0127] 6-4. Experimental Results Figure 25 shows the estimated signal and estimated image when two sound sources are arranged. Figure 26 shows the estimated signal and estimated image when four sound sources are arranged. The estimated signal by the proposed method shows the estimated signal for all elements in the target point cloud layer 720. In the image showing the estimated signal, the vertical axis is the element number and the horizontal axis is time, and the part with a large amplitude is shown as an outline. Looking at the outline of the signal, it was found that the more elements there are, the more accurately the signal can be estimated.

[0128] The estimated images obtained using the proposed method show the images transformed using the estimated signals. In all of the estimated images obtained using the proposed method, the characteristics of the sound source position are reflected, and similar to the general shape of the signal, a clearer image could be obtained with a larger number of elements.

[0129] As a comparative example, the estimated image obtained using the conventional method (Delay & Sum method) is shown. When using the conventional method, artifacts occurred between sound sources. In contrast, no artifacts were observed with the proposed method.

[0130] Figure 27 shows a comparison of the image quality of images estimated using the proposed method and those estimated using the conventional method (Delay & Sum method). In Figure 27, image quality was evaluated using the mean squared error (MSE) between the brightness of the estimated image and the brightness of the ground truth image. As a result, the proposed method showed significantly higher estimation accuracy than the conventional method for both the case with 32 sensors and the case with 128 sensors.

[0131] As described above, the weighting coefficient k includes coefficients related to amplitude and coefficients related to time. ｉBy using this method, even when the signal used in measurements requiring image reconstruction is a signal that takes time to propagate depending on the physical distance, it is possible to ensure measurement accuracy while solving at least one of the following (1) and (2): (1) Improving the degree of freedom of the physical distance between the sensor and the sensing object compared to conventional methods. (2) Arranging the sensors more sparsely than conventional methods.

[0132] 6-5. Industrial Applicability In the second embodiment, the signal was time-shifted, taking into account the wave propagation time, which is a characteristic of ultrasonic signals. Achieving ultrasonic imaging with a sparse sensor arrangement is essential for the practical application of flexible ultrasonic probes. When using 32 sensors, which is 1 / 8 the density of a dense rectangular array (16 × 16 = 256), the estimation accuracy was significantly higher than that of conventional methods. Therefore, the proposed method in this embodiment is considered to contribute to the practical application of flexible ultrasonic probes.

[0133] Although various embodiments of the present invention have been described above, these are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents.

[0134] 6. Modifications Section 6 describes modifications of this embodiment.

[0135] An embodiment of this design may be a program. This program is configured to cause a computer to execute each step of the information processing method of this design.

[0136] The control unit 210 writes (stores) and reads various data and information to the storage unit 220, but is not limited to this. For example, it may also use registers or cache memory within the control unit 210 to perform information processing for each activity.

[0137] The control unit 310 writes (stores) and reads various data and information to the storage unit 320, but is not limited to this. For example, it may also use registers or cache memory within the control unit 310 to perform information processing for each activity.

[0138] In this embodiment, a CPU is given as an example of the control unit 210 in the information processing device 200 and the control unit 310 in the terminal 300, but it is not limited to this. The control unit 210 and the control unit 310 may be a CPU, a graphics processing unit (GPU), a neural processing unit (NPU), a tensor processing unit (TPU), or a combination of these processors. In other words, the control unit 210 and the control unit 310 refer to one or more of the above processors, and the information processing method of this embodiment may be executed by having these processors cooperate.

[0139] The display information in this embodiment may be visual information such as a screen, image, icon, or message displayed on the display unit 330 of the terminal 300, or it may be rendering information for displaying such visual information on the display unit 330 of the terminal 300.

[0140] In the first embodiment, the heart was used as an example of the sensing target, and a method for measuring the membrane potential of the heart was described, but the method is not limited to this. For example, as described in the second embodiment, the information processing method of this embodiment can also be applied to ultrasound measurements.

[0141] Even when these modifications are adopted, the effects and advantages of this embodiment will still be exhibited. Furthermore, it is possible to combine this embodiment with its modifications, and with each other, as appropriate.

[0142] 7. The product may also be provided in any of the following forms:

[0143] (1) An information processing method comprising an acquisition step, a generation step, a conversion step, and an output step, wherein in the acquisition step, a sensor point cloud layer defined as a layer of point clouds representing the arrangement of a plurality of sensors, a target point cloud layer defined as a layer of point clouds representing the area to be sensed, and measurement data measured by each of the sensors are acquired, and a physical distance is defined between the sensor point cloud layer and the target point cloud layer, in the generation step, one or more intermediate point cloud layers defined as layers of point clouds located between the sensor point cloud layer and the target point cloud layer are generated, in the conversion step, each of the measurement data is propagated stepwise from the sensor point cloud layer to the target point cloud layer via the intermediate point cloud layer as a feature quantity for each point included in the intermediate point cloud layer, in the conversion step, each of the feature quantities propagated to the target point cloud layer is converted into an attribute value for each point in the target point cloud layer, and in the output step, the attribute value is output.

[0144] In this configuration, in measurements requiring image reconstruction, it is possible to ensure measurement accuracy while solving at least one of the following (1) and (2): (1) improving the degree of freedom of the physical distance between the sensor and the sensing object compared to conventional methods; and (2) arranging the sensors more sparsely than conventional methods. Furthermore, due to the simple configuration, the saved resources can be used for other core functions.

[0145] (2) The information processing method described in (1) above, wherein in the generation step, the intermediate point cloud layer is generated while increasing the density of the point cloud as the sensor point cloud layer moves toward the target point cloud layer.

[0146] In this configuration, even if there is a discrepancy between the number of points (sensors) included in the sensor point cloud layer and the number of points (measurement points) included in the target point cloud layer, the accuracy of the measurement can be ensured by applying an intermediate point cloud layer as a buffer.

[0147] (3) An information processing method according to (1) or (2) above, wherein in the generation step, when the position of the intermediate point cloud layer reaches the position of the target point cloud layer, the intermediate point cloud layer is generated such that the number of points included in the intermediate point cloud layer matches the number of points included in the target point cloud layer.

[0148] With this configuration, even with a sparse sensor arrangement, it is possible to acquire a number of attribute values ​​corresponding to the number of points (measurement points) in the target point cloud layer.

[0149] (4) An information processing method according to any one of (1) to (3) above, wherein in the generation step, when generating a new intermediate point cloud layer, new points are interpolated based on the position information of each point included in the previous layer and the position information of each point included in the target point cloud layer.

[0150] This configuration allows the number of points (measurement points) included in the final target point cloud layer to be matched.

[0151] (5) An information processing method according to any one of (1) to (4) above, wherein in the conversion step, the feature quantities of each point included in the previous layer are propagated to the newly generated intermediate point cloud layer.

[0152] In this configuration, the accuracy of the features of each point in each layer can be ensured by propagating the extracted features.

[0153] (6) An information processing method according to any one of (1) to (5) above, wherein in the conversion step, a convolutional neural network model is used to extract the measurement data of each point in the sensor point cloud layer and the feature quantities of each point in the intermediate point cloud layer.

[0154] In this configuration, the spatiotemporal distribution of physical quantities in real space can be estimated from multiple measurement data.

[0155] (7) An information processing method according to any one of (1) to (6) above, wherein in the conversion step, a first point is selected from the point cloud in the sensor point cloud layer or the intermediate point cloud layer; in the conversion step, the relative coordinates of the first point are calculated based on the coordinates of the first point and the coordinates of one or more second points adjacent to the first point; in the conversion step, a weight coefficient corresponding to the first point is calculated based on the relative coordinates and a mathematical model, the mathematical model being a model that shows the relationship between the relative coordinates of a certain point and the weight coefficient corresponding to that point; and in the conversion step, a feature quantity of the first point is extracted based on the weight coefficient and the measurement data or feature quantity of the second point.

[0156] In this configuration, the spatiotemporal distribution of physical quantities in real space can be estimated from multiple measurement data.

[0157] (8) The information processing method described in (7) above, wherein the weight coefficient includes a coefficient relating to the amplitude of the measurement data and a coefficient relating to time.

[0158] According to this embodiment, even when the signal used in a measurement requiring image reconstruction is a signal that takes time to propagate depending on the physical distance, it is possible to ensure measurement accuracy while solving at least one of the following (1) and (2): (1) Improving the degree of freedom of the physical distance between the sensor and the sensing object compared to conventional methods. (2) Arranging the sensors more sparsely than conventional methods.

[0159] (9) A method for generating a trained model, configured to use a computer to perform a creation step and a generation step, wherein the creation step creates a target dataset which is a dataset of attribute values ​​of a sensing target, the creation step creates a sensor dataset which is a dataset of coordinates where multiple sensors are arranged, the creation step creates a measurement dataset which is a dataset of measurement data measured at different heights by the multiple sensors, and the generation step generates a trained model based on the target dataset, the sensor dataset and the measurement dataset, wherein the trained model is a model that outputs the attribute values ​​in response to the coordinates of each sensor and the measurement data as inputs.

[0160] According to this embodiment, it is possible to provide technology using the information processing method of this embodiment.

[0161] (10) An information processing system comprising a control unit, wherein the control unit is configured to perform each step of the information processing method described in any one of (1) to (8) above.

[0162] In this configuration, in measurements requiring image reconstruction, it is possible to ensure measurement accuracy while solving at least one of the following (1) and (2): (1) improving the degree of freedom of the physical distance between the sensor and the sensing object compared to conventional methods; and (2) arranging the sensors more sparsely than conventional methods. Furthermore, due to the simple configuration, the saved resources can be used for other core functions.

[0163] (11) A program configured to cause a computer to perform each step of the information processing method described in any one of (1) to (8) above.

[0164] In this configuration, in measurements requiring image reconstruction, at least one of the following (1) and (2) can be solved while ensuring measurement accuracy: (1) The degree of freedom of the physical distance between the sensor and the sensing object can be improved compared to conventional methods. (2) Sensors can be spaced more sparsely than conventional methods. Furthermore, due to the simple configuration, the saved resources can be used for other core functions. Of course, this is not limited to these methods.

[0165] 100: Information processing system, 200: Information processing device, 210: Control unit, 211: Acquisition unit, 212: Generation unit, 213: Conversion unit, 214: Output unit, 215: Creation unit, 220: Storage unit, 250: Communication unit, 260: Communication bus, 300: Terminal, 310: Control unit, 320: Storage unit, 330: Display unit, 340: Input unit, 350: Communication unit, 360: Communication bus, 410: Sensor point cloud layer, 41 1: First point, 412: Second point, 413: Second point, 414: Second point, 415: Second point, 416: Second point group, 420: Target point cloud layer, 421: Attribute value, 430: Measurement data, 441: Intermediate point cloud layer, 442: Intermediate point cloud layer, 443: Intermediate point cloud layer, 510: Target dataset, 520: Sensor dataset, 530: Measurement dataset, 610: Sensor point cloud layer, 62 0: Target point cloud layer, 641: Intermediate point cloud layer, 642: Intermediate point cloud layer, 643: Intermediate point cloud layer, 701: Sensor, 702: Sensor, 703: Sensor, 704: Sensor, 705: Sensor, 706: Sensor, 707: Sensor, 708: Sensor, 710: Sensor point cloud layer, 711: First point, 712: Second point, 713: Second point, 714: Second point, 715: Second point, 716: Second point Group, 720: Target point cloud layer, 730: Imaging area, 731: Signal, 732: Signal, 733: Signal, 741: Midpoint point cloud layer, 742: Midpoint point cloud layer, 743: Midpoint point cloud layer, 760: Sound source, 761: Sound source, 762: Sound source, 763: Sound source, 770: Sensor, 780: Virtual sensor, 781: Virtual sensor, 782: Virtual sensor, 783: Virtual sensor, 784: Virtual sensor, 790: Image

Claims

1. An information processing method comprising: an acquisition step; a generation step; a conversion step; and an output step, wherein in the acquisition step, a sensor point cloud layer defined as a layer of point clouds representing the arrangement of a plurality of sensors, a target point cloud layer defined as a layer of point clouds representing the area to be sensed, and measurement data measured by each of the sensors are acquired, a physical distance is defined between the sensor point cloud layer and the target point cloud layer, in the generation step, one or more intermediate point cloud layers defined as layers of point clouds located between the sensor point cloud layer and the target point cloud layer, in the conversion step, each of the measurement data is propagated stepwise from the sensor point cloud layer to the target point cloud layer, via the intermediate point cloud layer, as feature quantities for each point included in the intermediate point cloud layer, in the conversion step, each of the feature quantities propagated to the target point cloud layer is converted into attribute values ​​for each point in the target point cloud layer, and in the output step, the attribute values ​​are output.

2. An information processing method according to claim 1, wherein in the generation step, the intermediate point cloud layer is generated while increasing the density of the point cloud as the sensor point cloud layer moves toward the target point cloud layer.

3. An information processing method according to claim 1, wherein in the generation step, when the position of the intermediate point cloud layer reaches the position of the target point cloud layer, the intermediate point cloud layer is generated such that the number of points included in the intermediate point cloud layer matches the number of points included in the target point cloud layer.

4. An information processing method according to claim 1, wherein in the generation step, when generating a new intermediate point cloud layer, new points are interpolated based on the position information of each point included in the previous layer and the position information of each point included in the target point cloud layer.

5. An information processing method according to claim 1, wherein the conversion step propagates the feature quantities of each point included in the previous layer to the newly generated intermediate point cloud layer.

6. An information processing method according to claim 1, wherein the conversion step involves using a convolutional neural network model to extract the measurement data for each point in the sensor point cloud layer and the feature quantities for each point in the intermediate point cloud layer.

7. An information processing method according to claim 1, wherein in the conversion step, a first point is selected from the point cloud in the sensor point cloud layer or the intermediate point cloud layer; in the conversion step, the relative coordinates of the first point are calculated based on the coordinates of the first point and the coordinates of one or more second points adjacent to the first point; in the conversion step, a weight coefficient corresponding to the first point is calculated based on the relative coordinates and a mathematical model; the mathematical model is a model that shows the relationship between the relative coordinates of a certain point and the weight coefficient corresponding to that point; and in the conversion step, a feature quantity of the first point is extracted based on the weight coefficient and the measurement data or feature quantity of the second point.

8. The information processing method according to claim 7, wherein the weighting coefficient includes a coefficient relating to the amplitude of the measurement data and a coefficient relating to time.

9. A method for generating a trained model, configured to use a computer to perform a creation step and a generation step, wherein the creation step creates a target dataset which is a dataset of attribute values ​​of a sensing target, the creation step creates a sensor dataset which is a dataset of coordinates where multiple sensors are arranged, the creation step creates a measurement dataset which is a dataset of measurement data measured at different heights by the multiple sensors, and the generation step generates a trained model based on the target dataset, the sensor dataset and the measurement dataset, and the trained model is a model that outputs the attribute values ​​in response to the coordinates of each sensor and the measurement data as inputs.

10. An information processing system comprising a control unit, wherein the control unit is configured to perform each step of the information processing method described in any one of claims 1 to 8.

11. A program configured to cause a computer to perform each step of the information processing method described in any one of claims 1 to 8.

Images