Dynamic tracking system

Abstract

A dynamic tracking system (1) comprises a contour detection unit (12), a video display unit (24), and a contour adjustment unit (25). The contour detection unit (12) detects a contour of an organism in frame images constituting a video, and performs a process for setting a plurality of contour points aligned along the contour for each frame image. The video display unit (24) displays, via a GUI (20), a video constituted by the frame images by overlaying the plurality of set contour points. The contour adjustment unit (25) is capable of adjusting the positions of the plurality of contour points in the displayed video via the GUI (20).

Description

Dynamic Tracking System

[0001] The present invention relates to a dynamic tracking system.

[0002] Patent Document 1 discloses an evaluation method that analyzes the video of an experimental animal and uses the activity and center-of-gravity trajectory of the experimental animal as evaluation indicators.

[0003] Japanese Patent Application Laid-Open No. 2018-143112

[0004] In the evaluation method disclosed in Patent Document 1, only the movement of the center of gravity of the experimental animal can be obtained, and changes in the shape of the experimental animal cannot be evaluated. The evaluation method disclosed in Patent Document 1 has limitations in the content that can be evaluated.

[0005] The present invention has been made in view of the above circumstances, and an object thereof is to provide a dynamic tracking system capable of more detailed evaluation of the dynamics of a living body.

[0006] The dynamic tracking system according to the present invention includes a contour detection unit that performs, for each frame image, a process of detecting the contour of a living body shown in the frame image constituting a video and setting a plurality of contour points arranged along the contour; a video display unit that displays, via a graphic user interface, a video composed of frame images in which the plurality of set contour points are superimposed; and a contour adjustment unit that can adjust the positions of the plurality of contour points in the displayed video via the graphic user interface.

[0007] It may also be provided with a dynamic feature calculation unit that calculates the features of the dynamics of the living body based on the plurality of contour points, and a storage unit that stores tracking data including the frame image, the time when the frame image was captured, data representing the contour of the living body detected in the frame image, and data representing the features of the dynamics of the living body calculated in the frame image.

[0008] The system may also include an image acquisition unit that sequentially acquires the frame images from the imaging device and corrects the frame images using image processing parameters, and a multithreaded control unit that controls the image acquisition unit, the contour detection unit, and the dynamic feature calculation unit so that processing is performed in parallel on a frame image basis.

[0009] The system may also include an analysis unit that performs analysis processing on the tracking data stored in the memory unit to generate biological analysis data, and an analysis unit that generates plot data representing the analysis data in a predetermined number of dimensional spaces.

[0010] The contour detection unit may assign identification information to each contour point, and the video display unit may display the contour points based on the identification information so that the corresponding contours can be identified.

[0011] The contour detection unit may, when multiple contours overlap in the frame image, assign identification information for each of the overlapping contours to the contour points of the overlapping portion.

[0012] The system may also include a region of interest adjustment unit that adjusts the region of interest within the frame image via a graphic user interface, and a contour detection unit that detects the contour of a living organism within the region of interest.

[0013] The region of interest adjustment unit may be capable of adjusting the external shape of the region of interest.

[0014] The region of interest adjustment unit may convert the image within the region of interest into an image of the living organism viewed in the direction normal to the plane on which the living organism is placed within the region of interest.

[0015] The contour detection unit may perform filtering or smoothing processing on the detected contours.

[0016] The contour detection unit may detect the plurality of contour points so that they are arranged at equal intervals along the contour.

[0017] The imaging device for imaging living organisms may also be equipped with an imaging adjustment unit that allows adjustment of imaging parameters during imaging via a graphical user interface.

[0018] The system may also include an image adjustment unit that allows the user to adjust the image processing parameters during video display via a graphic user interface.

[0019] The contour adjustment unit may be configured to allow adjustment of the detection parameters used by the contour detection unit for detecting the contour of a living organism while the video is being displayed.

[0020] According to the present invention, the dynamics of living organisms can be evaluated in more detail.

[0021] This is a block diagram showing the functional configuration of a motion tracking system according to an embodiment of the present invention. This is a diagram illustrating the perspective transformation of the region of interest. This is a diagram showing an example of an image within the region of interest. This is a diagram showing an example of detected contours and contour points. This is a diagram showing how the position of contour points is adjusted. This is a diagram showing an example of calculated motion features. This is a block diagram showing the hardware configuration of the motion tracking system of Figure 1. This is a flowchart of the flow of biological motion evaluation. This is a flowchart of the pre-processing. This is a flowchart of the motion tracking process. This is a flowchart of the contour detection process. This is a flowchart of the analysis process. C. This is a diagram showing an example of an Elegance contour image. C. This is a diagram showing the first example of the contour shape type of Elegance. C. This is a diagram showing the second example of the contour shape type of Elegance. C. This is a diagram showing the third example of the contour shape type of Elegance. C. This is a diagram showing the fourth example of the contour shape type of Elegance. C. This is a diagram showing the fifth example of the contour shape type of Elegance. This is a graph showing the detection results of Omega Turn. C. This is a two-dimensional heatmap showing the movement of the body surface of Elegance. C. This is a diagram showing an example of the time change in Elegance's energy consumption. This figure shows an example of the time change in the curvature of C. Elegance's body. This figure shows the time change in the contour of C. Elegance in three-dimensional space. This is a two-dimensional heatmap showing an example of mouse movement. This figure shows an example of a mouse trajectory. This figure shows an example of the time change in the energy consumption of mouse movement. This is a three-dimensional heatmap showing the frequency of locations of an aged mouse over a certain period of time. This is a three-dimensional heatmap showing the frequency of locations of a young mouse over a certain period of time. This figure shows an example of the movement of a young mouse. This figure shows an example of the movement of an aged mouse. This is a three-dimensional heatmap showing an example of mouse movement. This figure shows an example of the results of principal component analysis.

[0022] Embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the embodiments and drawings described below. In the embodiments described below, expressions such as “having,” “including,” or “containing” also include the meaning of “consisting of” or “composed of.”

[0023] The dynamic tracking system 1 according to this embodiment, shown in Figure 1, is a system for tracking the dynamics of living organisms, such as experimental animals. Dynamic tracking refers to determining the temporal changes in the shape or movement of an individual organism, i.e., a living organism. The dynamic tracking system 1 can track the dynamics of living organisms of various sizes. In this embodiment, we will explain using Caenorhabditis elegans, a type of nematode, as a relatively small living organism, and a mouse as a relatively large living organism. Hereinafter, Caenorhabditis elegans will be abbreviated as C. elegans. The dynamic tracking system 1 is not limited to these organisms; it is also possible to track the dynamics of organisms smaller than C. elegans, organisms larger than mice, and organisms of intermediate size.

[0024] As shown in Figure 1, the motion tracking system 1 comprises an imaging device 2 and a data processing device 3. The imaging device 2 captures video of a living organism. When the organism to be imaged is a mouse, for example, a camera is used as the imaging device 2. When the organism is C. elegans, for example, an electron microscope is used as the imaging device 2.

[0025] The imaging field of view of the imaging device 2 must be wide enough to track the dynamics of living organisms, particularly their movements. For example, when tracking the movement of a mouse in an open field arena, it is desirable that the imaging field of the imaging device 2 be set to include the mouse's range of movement. It is desirable that imaging parameters such as scaling (magnification), exposure time, frame rate, and focal position can be adjusted in the imaging device 2, but it may not be adjustable. It is also desirable, but not limited to, that the imaging device 2 has an illumination light source and that the brightness of the area being imaged can be adjusted. If the imaging device 2 has an illumination light source and the brightness of that illumination light source can be adjusted, the brightness of that illumination also becomes an imaging parameter. If the imaging device 2 can adjust lens distortion, the adjustment parameter for that lens distortion also becomes an imaging parameter.

[0026] Furthermore, multiple imaging devices 2 may be provided to image the living organism from multiple directions. In this case, it is desirable that the imaging parameters can be adjusted for each imaging device 2. The relative positional relationship between the imaging devices 2 (the relative positional relationship of each imaging field) can also be an imaging parameter. Through prior calibration, the relationship between the captured image and the actual size of the subject (physical scale coefficient) is determined in advance according to each settable imaging magnification, and when calculating dynamic features, the physical size (actual size) of the subject in the captured image can be determined from these relationships.

[0027] The data processing device 3 is an information processing device that processes video of living organisms captured by the imaging device 2. As described later, the data processing device 3 is realized by a computer executing a software program. A standard general-purpose notebook computer, i.e., a laptop personal computer (for example, an 8th generation computer with 8GB of memory), can be used as the computer. The software program that runs on the computer can be developed using, for example, various functions provided from the publicly known OpenCV® image processing library, but is not limited to this. The programming language used for software development can be, for example, Python®, but is not limited to this.

[0028] As shown in Figure 1, the data processing device 3 comprises an image processing unit 4 and a human-machine interface (hereinafter abbreviated as "MMI") 5. The image processing unit 4 performs image processing on the video captured by the imaging device 2. The MMI 5 is an interface with the user operating the computer, and is the part that displays the video and accepts user input.

[0029] As shown in Figure 1, the image processing unit 4 comprises an image acquisition unit 10, a region of interest identification unit 11, a contour detection unit 12, a dynamic feature calculation unit 13, a storage unit 14, an analysis unit 15, and a multithreaded control unit 16. The image acquisition unit 10 acquires a video of a living organism from the imaging device 2. The image acquisition unit 10 performs corrections to the video using image processing parameters. The region of interest identification unit 11 identifies a region of interest (ROI) within the video. The contour detection unit 12 detects the contours of living organisms in the frame images that make up the video. The dynamic feature calculation unit 13 calculates the dynamic features of the living organisms whose contours have been detected.

[0030] The storage unit 14 stores the frame images that make up the video, the time (timestamp) when the frame images were captured, the region of interest identified by the region of interest identification unit 11 in the frame images, the contour of the living organism detected in the frame images by the contour detection unit 12, and the dynamic features calculated in the frame images by the dynamic feature calculation unit 13. This data stored in the storage unit 14 is called tracking data. Tracking data is stored for each frame image. The analysis unit 15 performs various analyses based on the tracking data stored in the storage unit 14 to generate analysis data. The analysis data can be displayed on the MMI 5 and can be output externally.

[0031] The image acquisition unit 10, the region of interest identification unit 11, the contour detection unit 12, and the dynamic feature calculation unit 13 are each configured as independently operating threads. The multithreaded control unit 16 controls the execution of the image acquisition unit 10, the region of interest identification unit 11, the contour detection unit 12, and the dynamic feature calculation unit 13. When the image acquisition unit 10, the region of interest identification unit 11, the contour detection unit 12, and the dynamic feature calculation unit 13 complete their respective processes, they output a completion notification to the multithreaded control unit 16. When the multithreaded control unit 16 receives this completion notification, it outputs a processing start notification to the lower processing unit. In this way, the image processing unit 4 realizes so-called pipeline processing, in which the image acquisition unit 10, the region of interest identification unit 11, the contour detection unit 12, and the dynamic feature calculation unit 13 perform their respective processes in parallel for each frame image constituting the video. The multithreaded control unit 16 controls the image acquisition unit 10, the region of interest identification unit 11, the contour detection unit 12, and the dynamic feature calculation unit 13 to perform processing in parallel on a frame image basis.

[0032] The MMI5 comprises a graphic user interface (hereinafter abbreviated as GUI) 20, an imaging adjustment unit 21, an image adjustment unit 22, a region of interest adjustment unit 23, a video display unit 24, a contour adjustment unit 25, a dynamic feature adjustment unit 26, a memory adjustment unit 27, and an analysis adjustment unit 28.

[0033] GUI 20 is an interface with the user and primarily has the basic function of displaying data such as video output from the image processing unit 4, and the basic function of inputting user operations (operations specified by clicking on a pointing device, text input, etc.) to the image processing unit 4. The imaging adjustment unit 21 can command the imaging device 2 to start and end imaging via GUI 20. The imaging adjustment unit 21 can adjust the imaging parameters of the imaging device 2 when imaging a living organism via GUI 20. The image adjustment unit 22 can adjust the image processing parameters used for frame image correction performed by the image acquisition unit 10 via GUI 20. The region of interest adjustment unit 23 can adjust specific parameters for identifying the region of interest within the frame images constituting the video via GUI 20. The video display unit 24 displays a video via GUI 20, which is composed of an image in which the contour of a living organism and multiple contour points output from the contour detection unit 12 are superimposed on a frame image. The contour adjustment unit 25 can adjust the detection parameters used for contour detection in the contour detection unit 12 via the GUI 20. The dynamic feature adjustment unit 26 can set the calculation parameters used for calculating dynamic features in the dynamic feature calculation unit 13 via the GUI 20. The storage adjustment unit 27 adjusts the input and output of data recorded in the storage unit 14 via the GUI 20. The analysis adjustment unit 28 adjusts the analysis content performed by the analysis unit 15 via the GUI 20. The GUI 20 provides on-screen components such as sliders, checkboxes, radio buttons, and drop buttons. Various adjustments and controls are performed by inputting into these components through the operation of a pointing device or keyboard.

[0034] The following describes in more detail the functions of the imaging device 2, image acquisition unit 10, region of interest identification unit 11, contour detection unit 12, dynamic feature calculation unit 13, storage unit 14, and analysis unit 15.

[0035] The imaging device 2 performs imaging of a living organism according to the imaging parameters set by the imaging adjustment unit 21. Basically, the imaging magnification is set so that the range of movement of the living organism is contained within the imaging field of view, and the brightness is set so that the living organism can be clearly distinguished from other objects. The imaging adjustment unit 21 can adjust the imaging parameters of the imaging device 2 that is imaging the living organism during imaging via the GUI 20.

[0036] The image acquisition unit 10 sequentially acquires (captures) frame images that make up the video from the imaging device. When the image acquisition unit 10 captures frame images that make up the video input from the imaging device 2, it corrects the captured frame images. Image processing parameters set by the image adjustment unit 22 are used for this image processing. Such image processing parameters include, for example, an offset of the brightness value of each pixel to adjust the brightness of the frame image, a contrast value, a gamma value in gamma correction, and correction data to correct lens distortion. The image adjustment unit 22 can adjust the image processing parameters used to correct the frame images via the GUI 20 while the video is being displayed by the video display unit 24.

[0037] The region of interest identification unit 11 identifies the region of interest in the frame images of the video. The region of interest is set by the region of interest adjustment unit 23. The region of interest adjustment unit 23 can set any shape (e.g., rectangle or circle) as the outline of the region of interest. For example, when tracking the movement of a mouse in a Morris channel, it is desirable to set a circle as the shape that can be set. In addition, it is also possible to set the outline of the region of interest to an ellipse. The outline of the region of interest becomes one of the specific parameters for identifying the region of interest. The region of interest adjustment unit 23 can adjust the region of interest while the video is being displayed by the video display unit 24.

[0038] As shown in Figure 2, if the shape of the region of interest R is a rectangle, the region of interest adjustment unit 23 can specify four points RP at the four corners of the rectangle within the frame image F. In this case, the rectangle with the specified four points RP as its corners becomes the reference for the region of interest R. The region of interest adjustment unit 23 converts the image within the region of interest R of the frame image F into an image RI of the living organism viewed in the direction of the normal (basically vertical) of the plane on which the living organism is placed (for example, the ground in the area where the living organism can move) within the region of interest R. This conversion is performed by perspective transformation. Perspective transformation is performed using a perspective transformation matrix. This perspective transformation matrix becomes one of the specific parameters that identify the region of interest R. The perspective transformation matrix is ​​obtained by prior calibration. Note that coordinate transformation of the image may be performed by a transformation method other than perspective transformation. The specified points RP may be at least three points that are not collinear. This is because the region of interest R may be a triangle or a circle in addition to a rectangle.

[0039] For example, as shown in Figure 3, if living organisms T1 and T2 are captured in the image RI corresponding to the region of interest R, the contour detection unit 12 detects the contours C of living organisms T1 and T2 within the identified image RI, as shown in Figure 4. The number of detected contours C varies depending on the number of living organisms, and may be one or three or more. The contour detection unit 12 performs filtering or smoothing on the detected contours of living organisms T1 and T2. The threshold used for filtering or smoothing becomes one of the detection parameters. Such detection parameters include the edge detection threshold, Gaussian blur, blur kernel size, blur intensity, number of dilation cycles, and minimum contour area, which will be described later. The number of dilation cycles is the number of dilation cycles to enhance edges, and the minimum contour area is the threshold area for excluding contours C smaller than or equal to a specified area. The detected contours C are output to the video display unit 24 along with the frame image F. The video display unit 24 displays the frame image F in which the output contours C are superimposed.

[0040] As shown in Figure 4, the contour detection unit 12 generates a plurality of contour points CP arranged along contour C. The contour points CP are set to be arranged at equal intervals along contour C. The video display unit 24 displays a video composed of frame images F formed by superimposing the set plurality of contour points CP. The contour adjustment unit 25 can adjust the positions of the plurality of contour points CP in the displayed video via the GUI 20. For example, as shown in Figure 5, if there is a discrepancy between the detected contour C and the actual biological contour C', a discrepancy will also occur in the contour points CP. In this case, the contour adjustment unit 25 shifts the position of the contour points CP on the display screen onto contour C' based on user input, for example, from a pointing device. When the position of the contour points CP is shifted, the contour adjustment unit 25 inputs the position shift information to the contour detection unit 12, and the contour detection unit 12 corrects the position of the contour points CP so that the changed position of the contour points CP is on contour C'. Through this correction, the contour detection unit 12 aligns contour C with contour C'.

[0041] The detection parameters used to detect the contour of a living organism can be adjusted by the contour adjustment unit 25. These contour detection parameters include the line thickness of the contour C and the density of contour points CP (sample point density). Sample point density indicates the number of contour points CP evenly distributed from the contour C of the living organism. The sample point density is set to efficiently and precisely record the shape of the contour C and improve analysis accuracy. The sample point density is adjusted by operating a slider for adjusting the sample point density displayed on the display screen via the GUI 20. A higher number of points yields more detailed data, but increases the computational load. When the sample point density is set to 200, 200 contour points CP are evenly distributed across the entire contour C. The contour adjustment unit 25 can adjust the detection parameters used by the contour detection unit 12 for detecting the contour C while the video is being displayed. That is, these detection parameters can be adjusted in real time and flexibly changed according to the analysis target and experimental environment. This enables long-term continuous observation in changing environments and optimizes the balance between data accuracy and efficiency.

[0042] As shown in Figure 4, the contour detection unit 12 detects multiple contour points CP of multiple biological tissues T1 and T2 in the video, and assigns identification information to each of the detected contour points CP and corresponding contours C. The video display unit 24 displays the contour points CP so that the corresponding contours C can be identified based on the assigned identification information. For example, adjacent points CP can be displayed in different colors according to their corresponding contours C, or the identification information assigned to the contour points CP can be displayed. The contour detection unit 12 also assigns individual identification numbers to the contour points CP. For the second and subsequent frame images, the contour detection unit 12 refers to the positions of the multiple contour points CP in the previous frame image and assigns the identification numbers corresponding to the contour points CP in the previous frame image to the contour points CP in the current frame image. This makes it possible to track the time change in the position of the contour points CP.

[0043] The dynamic feature calculation unit 13 calculates the dynamic features of living organisms T1 and T2 based on, for example, a plurality of contour points CP. The dynamic feature calculation unit 13 calculates data representing the dynamic features of living organisms T1 and T2 based on a plurality of contour points CP on the contours C of living organisms T1 and T2 that are captured in the video detected by the contour detection unit 12 and adjusted by the contour adjustment unit 25. Examples of data representing dynamic features include the size, distance traveled, and speed of living organisms T1 and T2. Specifically, as shown in Figure 6, the data representing dynamic features includes the size and diameter of the smallest circle E that encloses the contours C of living organisms T1 and T2, the center position O of living organisms T1 and T2, the change in the center position O of living organisms T1 and T2 (velocity, acceleration), and the contour area.

[0044] The calculation of dynamic features is performed using calculation parameters. The calculation parameters include, for example, a physical scaling coefficient. The physical scaling coefficient is a coefficient for converting the pixel coordinates of a camera into physical units (e.g., millimeters). The physical scaling coefficient is used to accurately calculate physical measurements (e.g., the size of a biological T1, T2, the moving distance, the speed). The physical scaling coefficient is calibrated in advance to obtain the scale of the camera (how many millimeters one pixel corresponds to). If the physical scaling coefficient is 0.01 mm / pixel, it means that 100 pixels correspond to 1 mm. Such calculation parameters are set so that they can be confirmed and changed on the adjustment screen of the physical scaling coefficient displayed by the GUI 20.

[0045] In the storage unit 14, the frame image F of the video output from the dynamic feature adjustment unit 26, the time when the frame image F was captured, the region of interest R (image RI) of the frame image F specified by the region of interest specifying unit 11, the data representing the detected contour C in the frame image F detected by the contour detection unit 12 (e.g., the position of the contour point CP, etc.), and the data representing the dynamic features in the frame image F calculated by the dynamic feature calculation unit 13 are stored as tracking data for each frame image F in an associated manner. These tracking data can be stored, for example, as a file in CSV format, but are not limited thereto. The stored content of the storage unit 14 is controlled by the storage adjustment unit 27. For example, it is not necessary to store all the tracking data, and it is also possible to widen the time interval for storing the tracking data. In addition, the storage adjustment unit 27 can also automatically clean the tracking data stored in the storage unit 14.

[0046] The analysis unit 15 performs analysis processing on the tracking data stored in the memory unit 14 to generate analysis data, and generates display data that represents the analysis data in a predetermined number of dimensional spaces. This analysis is controlled by the analysis adjustment unit 28. The analysis adjustment unit 28 specifies the content to be analyzed. The content to be analyzed includes, for example, the time change of contour C. Furthermore, data representing the movement of the organism to be analyzed includes kinetic entropy, energy consumption, and randomness of movement based on the size, velocity, and distance traveled of the organism, and data representing the morphology of the organism to be analyzed includes the time change of shape based on contour points CP, bending frequency, curvature of the body, and contour area. The content to be analyzed is selected by the analysis adjustment unit 28.

[0047] Here, bending frequency is a quantification of the frequency with which organisms such as nematodes (C. elegans) and mice change the curvature of their bodies during movement. Bending frequency represents the number of significant bends or posture changes per unit time and serves as an indicator for evaluating the activity level and movement dynamics of organisms. Bending frequency is calculated as follows: For combinations of three consecutive contour points CP along contour C, the angle formed by the line segments connecting these contour points CP is calculated using trigonometric functions (e.g., arctangent), and the magnitude of the curvature of the overall contour C is calculated based on the sum of the angles at the consecutive contour points CP along contour C. Next, the magnitude of the curvature of the overall contour C is determined by a threshold to identify significant bends. For example, 45 degrees is set as this threshold. When the angle change exceeds the threshold, it is counted as a "significant bend." Furthermore, the number of significant bends per unit time is tallied. The number of significant bends in a predetermined time is the bending frequency. Analysis parameters such as thresholds necessary for this processing are adjusted by the analysis adjustment unit 28.

[0048] The analysis unit 15 performs analysis processing on data representing the characteristics of an individual's movement state to generate analysis data, and generates display data representing the analysis data in a predetermined number of dimensional spaces. Basically, the analysis adjustment unit 28 designates the specified analysis content and the necessary data, and the analysis unit 15 reads out the tracking data related to the designated data from the storage unit 14. Then, the analysis adjustment unit 28 designates which data corresponds to each axis of the graph or heat map to be displayed. The analysis unit 15 performs calculations related to the analysis content based on the designated data based on the designation, obtains the calculation result, and plots the calculation result on the graph or heat map. The generated display data is, for example, a graph showing the contour C and its changes, a graph showing the temporal changes of the movement characteristics, a two-dimensional or three-dimensional heat map of the movement characteristics, a graph showing various analysis results such as the principal component analysis of the movement characteristics. The generated display data is displayed by the analysis adjustment unit 28. In addition, the analysis unit 15 can output the generated analysis data as a file in a predetermined format.

[0049] [Hardware Configuration] The data processing device 3 that constitutes the movement tracking system 1 shown in FIG. 1 is realized, for example, by a computer having the hardware configuration shown in FIG. 7 realizing a software program. Specifically, the data processing device 3 includes a CPU (Central Processing Unit) 31 that controls the entire device, a main memory 32 that operates as a work area of the CPU 31, an external memory 33 that stores a program 38 executed by the CPU 31, an operation input unit 34, a display 35, a communication interface (I / F) 36, and an internal bus 37 that connects these.

[0050] The CPU 31 realizes the functions of the movement tracking system 1 by executing the program 38 read into the main memory 32.

[0051] The main memory 32 consists of RAM (Random Access Memory), etc. As mentioned above, the program 38 executed by the CPU 31 is loaded into the main memory 32 from the external memory 33. The main memory 32 is also used as the CPU 31's work area (temporary data storage area).

[0052] The external memory 33 consists of non-volatile memory such as flash memory or a hard disk. The external memory 33 has a program 38 pre-stored in it for the CPU 31 to execute.

[0053] The operation input unit 34 consists of devices such as a keyboard and a mouse, and an interface device that connects these devices to the internal bus 37.

[0054] The display 35 is composed of a display device such as a liquid crystal monitor or an organic EL (Electro-Luminescence) display.

[0055] The communication interface 36 is an interface for sending and receiving data with external devices. For example, captured images from the imaging device 2 are input via the communication interface 36.

[0056] The functions of the motion tracking system 1 can be implemented in a computer system consisting of one or more computers, each including one or more processors and one or more storage devices, including non-temporary storage media. Multiple computers communicate with each other via a communication network to implement the functions of the motion tracking system 1. For example, some of the functions of the motion tracking system 1 may be implemented in one computer, while other functions may be implemented in other computers.

[0057] [Operation of the Dynamic Tracking System] Next, the flow of evaluating the dynamics of a living organism using the dynamic tracking system 1 will be explained. In this series of operations, as shown in Figure 8, first, the dynamic tracking system 1 performs pre-processing (step S1). In this pre-processing, various parameters are calibrated prior to dynamic tracking.

[0058] In the preprocessing stage, as shown in Figure 9, the imaging adjustment unit 21 adjusts the imaging parameters (step S11). Here, imaging is performed by the imaging device 2, and while viewing the displayed image, the scaling (magnification), exposure time, frame rate, focal position, and illumination brightness are adjusted. The magnification is adjusted so that the range of motion of the living organism fits within the imaging field of view of the imaging device 2. The exposure time and illumination brightness are set so that the living organism is easily distinguishable from other objects. The focal position is adjusted so that the image of the living organism does not become blurred. The frame rate is set to a speed that can detect changes in the movement and shape of the living organism.

[0059] Next, the image adjustment unit 22 adjusts the image processing parameters (step S12). Here, the luminance value offset of each pixel in the image, the γ value in γ correction, and correction data for correcting lens distortion are set. The correction data for correcting lens distortion is calculated based on the positional shift of the markers in the captured image, after the imaging device 2 captures the markers with markers arranged on the imaging surface within the imaging field of view to adjust these parameters.

[0060] Next, the region of interest adjustment unit 23 sets specific parameters (step S13). As mentioned above, the specific parameters include the outline of the region of interest R and the transmission transformation matrix used for the perspective transformation of the region of interest R. The transmission transformation matrix is ​​calculated based on the positional relationship of the arrangement of the imaged markers, as described above.

[0061] Next, the contour adjustment unit 25 adjusts the detection parameters (step S14). As mentioned above, the detection parameters include, for example, a threshold used in filtering or smoothing, the line thickness of contour C, and the sample point density. The contour adjustment unit 25 sets the threshold, the line thickness of contour C, the sample point density, etc.

[0062] The dynamic feature adjustment unit 26 adjusts the calculation parameters (step S15). The calculation parameters include a physical scaling coefficient. The physical scaling coefficient is determined based on the size of the marker in the captured image, which is captured by the imaging device 2 with a marker displaying the actual physical scale.

[0063] Once the preprocessing is complete, the system returns to Figure 8, and the motion tracking system 1 performs motion tracking (step S2). In the motion tracking process, under the control of the multithreaded control unit 16, the image acquisition unit 10, the region of interest identification unit 11, the contour detection unit 12, and the motion feature calculation unit 13 perform processing in parallel, as shown in Figure 10.

[0064] First, let's explain the processing of the image acquisition unit 10. The image acquisition unit 10 waits until a frame image is input (step S21; No). Once a frame image is input (step S21; Yes), the image acquisition unit 10 receives a change command from the image adjustment unit 22 to change the image processing parameters and determines whether or not to change the image processing parameters (step S22). If it is determined that the parameters should be changed (step S22; Yes), the image acquisition unit 10 changes the image processing parameters (step S23).

[0065] If no change command is input (Step S22; No), or after Step S23 is completed, the image acquisition unit 10 corrects the frame image using the set image processing parameters (Step S24). Subsequently, the image acquisition unit 10 outputs the corrected image to the region of interest identification unit (Step S25). Next, the image acquisition unit 10 determines whether or not an instruction to terminate processing has been input (Step S26). If no instruction to terminate processing has been input (Step S26; No), the image acquisition unit 10 waits for a frame image input again (Step S21; No). If an instruction to terminate processing has been input (Step S26; Yes), the image acquisition unit 10 terminates processing.

[0066] Next, the processing of the region of interest identification unit 11 will be explained. The region of interest identification unit 11 waits until the corrected image is input (step S31; No). Once the corrected image is input (step S31; Yes), the region of interest identification unit 11 determines whether or not a change command to change the specific parameters has been input (step S32). If it is determined that a change is to be made (step S32; Yes), the region of interest identification unit 11 changes the specific parameters (step S33). This change can be performed by specifying four points RP on the video using a pointing device, as described above, but it can also be done by inputting coordinate values ​​using a keyboard.

[0067] If no command to change the region of interest R has been input (step S32; No), or after step S33 is completed, the region of interest identification unit 11 identifies the set region of interest R (step S34). At this time, the image of the region of interest R is converted into an image RI (see Figure 2) by perspective transformation or the like. Next, the region of interest identification unit 11 outputs the identified image in which the region of interest R has been identified to the contour detection unit 12 (step S35). Next, the region of interest identification unit 11 determines whether or not to terminate the process (step S36). If no command to terminate the process has been input (step S36; No), the region of interest identification unit 11 waits for input of the corrected image (step S31; No). If a command to terminate the process has been input (step S36; Yes), the region of interest identification unit 11 terminates the process.

[0068] Next, the processing of the contour detection unit 12 will be described. The contour detection unit 12 waits until it receives a identified image (image RI; see Figure 2) in which the region of interest R has been identified (step S41; No). When the identified image is received (step S41; Yes), the contour detection unit 12 determines whether or not a change command to change the detection parameters has been received (step S42). If it is determined that the parameters should be changed (step S42; Yes), the contour detection unit 12 changes the detection parameters (step S43).

[0069] If no command to change the detection parameters has been input (step S42; No), or after step S43 is completed, the contour detection unit 12 executes a contour detection processing subroutine (step S44).

[0070] In the contour detection processing subroutine, as shown in Figure 11, first, the contour detection unit 12 performs a grayscale conversion (step S60). This converts the identified image (image RI) from a color image to a grayscale image represented only by shades of color. Next, the contour detection unit 12 performs a blurring process (step S61). The blurring process converts the grayscale image into an image with smoothly changing brightness using, for example, a Gaussian distribution. Here, for example, Gaussian blurring and blurring processing are performed using the kernel size and blurring intensity set as detection parameters. Next, the contour detection unit 12 performs edge detection (step S62). For edge detection, for example, the Canny edge detection algorithm is applied. However, the edge detection method is not limited to the Canny edge detection method. Next, the contour detection unit 12 generates a mask image based on a set threshold (step S63). At this time, the mask image may be subjected to a set number of dilation processes. Next, the contour detection unit 12 performs filtering to exclude contours C with an area smaller than the minimum contour area (step S64). This excludes small contour lines that do not constitute the contour C of a living organism.

[0071] The contour detection unit 12 generates contour points CP, which are multiple points on the detected contour C (step S65). The contour points CP are generated by adaptive sampling so that they are equally spaced along the contour C. Next, the contour detection unit 12 assigns a tracking number to each contour point CP. If there are multiple detected contours C, each contour C is assigned a different tracking number (step S66). For example, as shown in Figures 3 to 5, if there are two living organisms T1 and T2, the contour C and contour point CP are assigned tracking numbers T1 and T2, respectively. In addition, each individual contour point CP is also assigned a unique tracking number. After step S66 is executed, the contour detection processing subroutine is terminated.

[0072] Returning to Figure 10, the contour detection unit 12 then outputs the detected image in which contour C has been detected to the dynamic feature calculation unit 13 (step S45). Next, the contour detection unit 12 determines whether or not to terminate the process (step S46). If no command to terminate the process has been input (step S46; No), the contour detection unit 12 waits again for input of the identified image (step S41; No). If a command to terminate the process has been input (step S46; Yes), the contour detection unit 12 terminates the process.

[0073] The processing of the dynamic feature calculation unit 13 will now be described. The dynamic feature calculation unit 13 waits until a detected image in which contour C has been detected is input (step S51; No). Once a detected image is input (step S51; Yes), the dynamic feature calculation unit 13 determines whether or not a change command has been input to change the calculation parameters in the dynamic feature adjustment unit 26 (step S52). If it is determined that a change is to be made (step S52; Yes), the dynamic feature calculation unit 13 changes the calculation parameters (step S53).

[0074] If no command to change the calculation parameters has been input (step S52; No), or after step S53 is completed, the dynamic feature calculation unit 13 performs the dynamic feature calculation process (step S55). As shown in Figure 6, for example, the data representing the calculated dynamic features includes the size and diameter of the smallest circle E that encloses the contours C of the living organisms T1 and T2, the center position O of the living organisms T1 and T2, the change in the center position O of the living organisms T1 and T2 (velocity, acceleration), and the contour area.

[0075] Next, the dynamic feature calculation unit 13 generates tracking data including the video frame image F, the time the frame image F was captured, the region of interest R, the contour C, and dynamic features, and stores them in the storage unit 14 (step S55). Subsequently, the dynamic feature calculation unit 13 determines whether or not to terminate the process (step S56). If no command to terminate the process is input (step S56; No), the dynamic feature calculation unit 13 waits again for input of the identified image (step S51; No). If a command to terminate the process is input (step S56; Yes), the dynamic feature calculation unit 13 terminates the process.

[0076] Furthermore, while this motion tracking process is being performed, the imaging adjustment unit 21 can change the imaging parameters as needed.

[0077] Returning to Figure 8, the motion tracking system 1 waits until an analysis command for the tracking data is input (Step S3; No). Once the analysis command is input (Step S3; Yes), the analysis unit 15 of the motion tracking system 1 performs the analysis (Step S4).

[0078] In the analysis process, as shown in Figure 12, first, the analysis unit 15 obtains the specified analysis type from the analysis adjustment unit 28 (step S80). The analysis type includes the content to be analyzed, the display format, etc. Next, the analysis unit 15 reads the tracking data necessary for the analysis of the specified type from the storage unit 14 (step S81). Next, the analysis unit 15 generates analysis data based on the read tracking data (step S82). Next, the analysis unit 15 generates plot data that represents the analysis data in a predetermined number of dimensional space based on the analysis data (step S83). Next, the analysis unit 15 outputs the plot data to the analysis adjustment unit 28 or externally (step S84). The analysis adjustment unit 28 displays the plot data as a graph or the like, plotting the analysis data in a predetermined number of dimensions, two-dimensional or three-dimensional space. When outputting externally, the plot data is output in the specified file format. After step S84 is completed, the analysis unit 15 terminates the process. The analysis process by the analysis unit 15 can be executed multiple times for each analysis target.

[0079] Returning to Figure 8, after the analysis process in step S4 is completed, the dynamic tracking system 1 completes the series of dynamic evaluations of the living organism.

[0080] [Analysis Examples] The dynamic tracking system 1 enables various analyses of living organisms. Several analysis examples are shown below.

[0081] The dynamic tracking system 1 allows for highly accurate analysis of morphological and motor changes in living organisms. Therefore, the dynamic tracking system 1 can also be used for screening to quantitatively evaluate the effects of drugs or chemical substances on living organisms. For example, by analyzing dynamic characteristics such as morphological changes, curvature, muscle contraction / relaxation cycles, movement speed, energy consumption, and kinetic entropy before and after administration of drugs or functional compounds using the dynamic tracking system 1, it becomes possible to evaluate muscle motor dysfunction, anti-aging-like phenomena, and neurophysiological changes.

[0082] In addition to these features, the dynamic tracking system 1 can also be used as a metabolic screening device to analyze metabolic activity, mitochondrial dysfunction, nutritional response, and the effects of metabolic enhancers. By using the dynamic tracking system 1 to quantify changes in energy consumption and exercise endurance in model organisms such as nematodes, fruit flies, and mice, the effects of metabolic diseases, obesity, and diabetes-related drugs can be easily detected.

[0083] Furthermore, the dynamic tracking system 1 is also applicable to neurobehavioral evaluation. For example, by quantifying exploratory behavior, social behavior, memory and learning behavior, and depressive-like behavior using dynamic indicators (velocity, frequency of direction changes, area of ​​stay, motor randomness, etc.) with the dynamic tracking system 1, it becomes possible to analyze neurodegenerative diseases, psychiatric disorders, or pharmacological effects.

[0084] Furthermore, the dynamic tracking system 1 is also useful for evaluating toxic effects caused by chemical substances and environmental factors. Because the dynamic tracking system 1 allows for early detection of neurotoxicity, myotoxicity, and developmental toxicity as abnormalities in movement patterns and morphological changes, it is expected to have applications in safety testing and environmental toxicity monitoring.

[0085] Furthermore, the dynamic tracking system 1 can also be applied to developmental and regenerative biological research. By tracking morphological changes during embryonic development, organogenesis, tissue regeneration, and wound healing, it can be used to evaluate regeneration promoters and cell growth factors.

[0086] Furthermore, the dynamic tracking system 1 can non-invasively detect muscle contraction function, muscle fatigue, and neuromuscular junction dysfunction. For this reason, the dynamic tracking system 1 is also useful in research on muscle diseases, sarcopenia, and age-related decline in motor function. In addition, by analyzing circadian rhythms and sleep-wake rhythms through long-term observation, the dynamic tracking system 1 can also be applied to the evaluation of sleep disorders and fatigue recovery.

[0087] As an application to the field of food and nutrition, the dynamic tracking system 1 can be used to analyze the effects of ingesting functional food components and nutritional supplements on the body's exercise patterns and energy metabolism, and to quantitatively evaluate effects such as anti-fatigue, antioxidant, anti-inflammatory, or cognitive function improvement. Furthermore, the dynamic tracking system 1 can also be applied to "gut-brain axis" research, which analyzes the relationship between the gut environment and behavioral changes.

[0088] Furthermore, the dynamic tracking system 1 can be used to determine the viscoelasticity and deformation response of living organisms from the displacement information of contour points, and can be used for evaluating the mechanical properties of living tissues, analyzing external force responses, or mechanically analyzing cell movement. This also makes it possible to quantify biorheological properties and evaluate tissue sclerosis diseases.

[0089] Furthermore, if the analysis unit 15 of the dynamic tracking system 1 is equipped with machine learning or deep learning algorithms to automatically classify disease-specific patterns from the obtained multidimensional dynamic data, it can be applied to the early diagnosis and prediction of aging, neurological diseases, metabolic diseases, muscle diseases, and other conditions. Thus, the dynamic tracking system 1 is extremely useful as a high-precision screening and quantitative analysis device in a variety of biomedical fields, including drug screening, anti-aging compound evaluation, neurophysiological analysis, and metabolic, toxicological, regenerative, nutritional, and mechanical research.

[0090] As an example of various analyses of living organisms, Figure 13 shows an example of a contour image of C. elegans obtained by the motion tracking system 1. As shown in Figure 13, the motion tracking system 1 makes it possible to display the contours of multiple C. elegans plants in a way that allows them to be distinguished from one another, for example, by color coding or shades of color.

[0091] Figures 14A to 14E show the types of contour shapes of C. elegans obtained by the dynamic tracking system 1. The contour shapes in Figures 14A, 14B, 14C, and 14D are C-type, S-type, hook-type, and coil-type. These shapes indicate that C. elegans is experiencing stress, seizures, neurological defects, exhibiting escape reactions, or aging. The shape shown in Figure 14E is called the omega turn. The omega turn is one of the contour shapes that appears when C. elegans is performing normal movements. The dynamic tracking system 1 detected the percentage of time that C. elegans exhibited the omega turn in C. elegans with the fard-1 gene knocked out and in wild-type (Wt) C. elegans. As shown in Figure 14F, C. elegans with the fard-1 gene knocked out... In C. elegans, the proportion of time spent performing omega turns was reduced compared to the wild type (Wt) (N=3, p<0.01). Furthermore, when C. elegans with the fard-1 gene knocked out was treated with sPls (5 μg / ml), the proportion of time spent performing omega turns increased (N=3, p<0.05).

[0092] Figure 15 shows a two-dimensional heatmap illustrating the movement of the body surface of C. elegans, as analyzed by the analysis unit 15 of the dynamic tracking system 1. Each point in this two-dimensional heatmap represents a contour point CP of C. elegans. In this two-dimensional heatmap, contour points CP detected over a certain period of time are superimposed and displayed, and areas with large variations in the point cloud represent areas with large movements. This two-dimensional heatmap allows for the analysis of how the body surface of C. elegans is moving.

[0093] Figure 16 shows an example of the time-dependent change in energy consumption of C. elegans. Figure 17 shows an example of the time-dependent change in the curvature of the C. elegans' body. This data can be used to evaluate the motor skills of C. elegans, including whether its movements are normal or abnormal. In addition, the time-dependent changes in contour area and straightness can also be displayed using similar graphs.

[0094] Furthermore, as shown in Figure 18, it is also possible to show the temporal changes in the contour of C. elegans in three-dimensional space. The X and Y axes of this graph represent the two-dimensional position of the area in which C. elegans can move, in units of mm. The Z axis represents the passage of time (seconds). In this map, the temporal changes in the morphology and position of C. elegans are represented by changes in grayscale, or more accurately, changes in color (shades of color in Figure 18). According to this graph, it is possible to observe and analyze the morphological changes of C. elegans and track subtle structural changes over time. In other words, the analysis supported by this dynamic tracking system 1 can precisely analyze the complex behavioral patterns and migration routes of organisms, supporting behavioral research. Dynamic tracking system 1 visualizes the morphology of living organisms in a temporal and spatial dimension, providing unprecedented insights into biological research. Dynamic tracking system 1 can be an extremely valuable tool in a wide range of fields, including developmental biology, aging research, stress analysis, and disease modeling. These features enable precise morphological tracking and behavioral analysis that surpasses conventional imaging techniques, forming a crucial foundation for patentability. This system represents cutting-edge technology applicable to research on organisms such as C. elegans, including pharmaceutical research, genetic research, and environmental stress testing.

[0095] Furthermore, for mice, the dynamic tracking system 1 can, for example, obtain a two-dimensional heatmap (Figure 19) showing their movements. The two-dimensional heatmap shown in Figure 19 actually visualizes the movement density of mice moving within the exploration area. The X and Y axes represent the spatial dimensions (in millimeters) of the exploration area, and in this heatmap, the number of times (frequency) the mouse was in a particular location is shown by the shades of grayscale, or more accurately, by the gradient of color intensity. Darker areas indicate areas with high activity, while lighter areas indicate areas with little or no activity. Such data can be used in neuroscience research to analyze and evaluate the behavioral patterns of subjects. Also, by analyzing areas with high and low exploration, it is possible to quantify how thoroughly the mouse was able to explore the environment. This is an important indicator in the study of anxiety levels and curiosity.

[0096] Furthermore, the motion tracking system 1 makes it possible to analyze the movement trajectory of a mouse over a certain period of time (Figure 20A) and the amount of energy consumed by the mouse calculated from its movements over a certain period of time (Figure 20B).

[0097] Figure 21A shows a 3D heatmap illustrating the movement of an aged mouse over a certain period of time, with the mouse's outline superimposed at each time point. Figure 21B shows a 3D heatmap illustrating the movement of a young mouse over a certain period of time, with the mouse's outline superimposed at each time point. Comparing Figures 21A and 21B reveals a clear difference in movement between the aged mouse and the young mouse, with the aged mouse often staying in the four corners of the rectangular area.

[0098] Figure 22A shows the movement of a young mouse, with sampled contours superimposed over a set period of time, while Figure 22B shows the movement of an older mouse, with sampled contours superimposed over a set period of time. Comparing Figures 22A and 22B, it can be seen that the young mouse's movements are random, while the older mouse moves in circles around the outer edge of the region, repeating the same behavior.

[0099] In addition, as shown in Figure 23, a three-dimensional heatmap representing the movement of an aging mouse can be generated with the vertical axis representing time, and as shown in Figure 24, it is also possible to perform principal component analysis on the two dynamic features obtained through dynamic tracking, using PC1 axis and PC2 axis, and display the analysis results. In Figure 24, the two arrows represent the first principal component and the second principal component, respectively.

[0100] The dynamic tracking system 1 can simultaneously track multiple mice within a region of interest R, accurately tracking them without misidentification of identification information. This makes it possible to accurately track multiple mice in an open field. In particular, it reduces stress on mice and allows for more accurate detection of their behavioral patterns because it eliminates the need to place conspicuous marks on them during experiments. This function is extremely useful in neurobiological studies evaluating the mental state of mice and in drug screening.

[0101] For example, the positions of two mice within a given area can be represented using a 3D heatmap. The X and Y axes represent the spatial position (cm) of the mice, and the Z axis represents the frequency of occurrence at a particular location. This 3D heatmap visualizes the density of mouse positions and movements, allowing for the prediction and analysis of social behaviors (e.g., social interaction and aggression) by identifying areas where both subjects frequently occupy the same or overlapping positions. Furthermore, areas showing high frequencies on the Z axis indicate locations where activity is increasing, and can be estimated to be areas where important interactions or habitual behaviors between the two subjects are occurring.

[0102] [Motivation and Depression Assessment] The movement patterns analyzed can be correlated with motivational states and depressive-like behaviors. For example, a decrease in mouse exploration or staying in a specific area suggests decreased motivation or depressive-like behavior.

[0103] [Spatial Memory and Cognitive Function] Tracking the movement density of mice over time can reveal patterns associated with spatial memory and cognitive abilities. If a mouse frequently revisits a particular area, it indicates that the mouse remembers a habit, preference, or a reward or safe zone.

[0104] [Effects of Drugs and Treatments] By comparing heatmaps before and after drug administration, the effects can be quantitatively evaluated. For example, if exploration increases or movement patterns become more balanced after antidepressant administration, it can demonstrate the effectiveness of the drug administration.

[0105] [Application to Scoring Behavioral Patterns] The dynamic tracking system 1 is highly effective for scoring the behavior of living organisms based on quantitative indicators. Activity level: The overall activity level can be scored by summing the movement density across the entire grid. Exploration zones: Preferences and avoidance behaviors can be scored by identifying high-density zones. Behavioral indicators: Scores such as "exploration index," "depression index," and "motivation score" can be calculated from movement patterns according to the experimental objective.

[0106] [Adaptation to the Latest Technologies in Behavioral Neuroscience] This heatmap visualization analysis can be applied as a cutting-edge technology in behavioral neuroscience. High-resolution behavioral mapping: It can provide spatial and temporal resolution to understand target behavior within the experimental setting. Automated scoring: It can be integrated with machine learning algorithms to automate behavioral scoring. This reduces the occurrence of human error and bias in the analysis results. Drug screening: By analyzing changes in movement patterns, the effects of drugs can be rapidly evaluated. Data integration: A comprehensive understanding of behavioral changes can be obtained by combining it with other data modalities such as physiological measurements (e.g., heart rate and stress hormone levels). Overall, the dynamic tracking system 1 is a groundbreaking addition to behavioral neuroscience research, accurately quantifying target behavior and providing high-resolution insights. It can be a crucial tool in both basic and clinical research for studies to understand the impact of new drugs and treatments on behavioral patterns.

[0107] [Summary] As described in detail above, the motion tracking system 1 according to this embodiment detects not only the center of gravity of the living organisms T1 and T2, but also the contour C and multiple contour points CP, so the motion of the living organisms T1 and T2 can be evaluated in more detail.

[0108] According to the motion tracking system 1 of this embodiment, detailed biological motion characteristics can be calculated based on a plurality of detected contour points CP. Furthermore, since tracking data including video frame images F, time, data representing the contour C of the living organism, and data representing the dynamic characteristics of the living organism are stored, more detailed analysis can be performed later based on the tracking data.

[0109] Furthermore, according to the motion tracking system 1 of this embodiment, the image acquisition unit 10, the region of interest identification unit 11, the contour detection unit 12, and the motion feature calculation unit 13 are controlled to perform processing in parallel each time a frame image F is acquired. This enables pipeline-like image processing during imaging, thereby reducing processing time. In addition, multi-thread control reduces the risk of motion tracking being interrupted.

[0110] The dynamic tracking system 1 according to this embodiment includes an analysis unit 15 that performs analysis processing on tracking data stored in a memory unit 14 to generate analysis data of a living organism, and generates plot data that represents the analysis data in a predetermined number of dimensional spaces. Since a wide variety of analysis processes can be employed in the analysis unit 15, it becomes possible to analyze various living organisms with various analysis content.

[0111] In the motion tracking system 1 according to this embodiment, the contour detection unit 12 assigns identification information to each contour C corresponding to the detected contour point CP, and the video display unit 24 displays the contour C in a way that allows each biological organism to be distinguished according to the identification information. In this way, it becomes easy to distinguish between biological organisms T1 and T2 (see Figure 2) displayed in the image.

[0112] When multiple overlapping contours C exist in the frame image F, the contour detection unit 12 assigns identification information for each of the overlapping contours C to the contour point CP of the overlapping portion. This prevents the contour point CP from being associated only with contours C that are not the contours C it originally corresponds to.

[0113] According to the motion tracking system 1 of this embodiment, the contour C of a living organism is detected within the region of interest R of the frame image F. In this way, since it is only necessary to detect the contour C of a living organism within the region of interest R specified by the user, the time required for detection can be reduced.

[0114] According to the motion tracking system 1 of this embodiment, the region of interest adjustment unit 23 can set any shape as the outer shape of the region of interest R. In this way, a region of interest suitable for the outer shape of the movement range of a living organism can be set.

[0115] According to the motion tracking system 1 of this embodiment, filtering or smoothing processing is performed on the generated contour C. In this way, noise components contained in the contour C are removed, and the contour C can be detected with high accuracy. For example, reliable contour detection becomes possible regardless of changes in lighting or contrast.

[0116] According to the dynamic tracking system 1 of this embodiment, the contour detection unit 12 detects a plurality of contour points CP arranged at equal intervals along the contour C. This makes it easier to identify which contour a contour point CP belongs to when multiple contours C overlap.

[0117] According to the motion tracking system 1 of this embodiment, imaging parameters, image processing parameters, parameters for specifying the region of interest R, parameters for detecting contours C, and parameters for calculating motion features can be changed during imaging. This allows for flexible modification of various parameters in response to environmental changes, enabling observation of the motion of living organisms over long periods of time in a changing environment. This optimizes the balance between data accuracy and efficiency.

[0118] The imaging parameters, image processing parameters, region of interest R specification parameters, contour C detection parameters, and dynamic feature calculation parameters are adjusted in this order. This allows for precise adjustment while the parameters used in upstream processing are already adjusted, enabling highly accurate contour detection and dynamic feature calculation.

[0119] According to the dynamic tracking system 1 of this embodiment, the time interval of the tracking data recorded for each frame image F can be adjusted. This prevents unnecessary data from being stored in the storage unit 14, thus preventing the required storage capacity from becoming excessively large.

[0120] Furthermore, in the dynamic tracking system 1 according to this embodiment, information on multiple contour points CP may be stored only, without storing information on the entire detected contour C. In this way, what information to store as contour information can be arbitrarily selected.

[0121] According to the motion tracking system 1 of this embodiment, before tracking the motion of a living organism, calibration is performed to adjust various parameters such as lens distortion of the imaging device 2. The adjustment values ​​of the various parameters adjusted by these calibrations can be stored for each imaging device 2. When performing motion tracking by imaging with that imaging device 2 thereafter, the stored adjustment values ​​of the various parameters can be read and set. This eliminates the need to perform calibration again. If the adjustment values ​​of the parameters calibrated for each imaging device 2 are digitized, even if the type of living organism being observed changes, the data can be read and the adjustment values ​​of the parameters can be switched, eliminating the need to perform calibration each time.

[0122] Furthermore, the contour detection process, dynamic feature extraction process, etc., described above are not limited to those mentioned above. A wide variety of image processing algorithms can be employed.

[0123] In the above embodiment, mice or nematodes are used as the target organism. However, other animals may be used as the target organism. It can be used primarily for research purposes in animal behavior, pharmacological testing, ecological monitoring, and neuroscience. Furthermore, by adjusting parameters such as brightness, blur, and threshold, it is possible to analyze the behavior and morphology of diverse organisms such as aquatic organisms (e.g., zebrafish), insects (e.g., fruit flies, ants), small birds, amphibians, and reptiles under various environmental conditions.

[0124] Furthermore, the organism whose dynamics are tracked by the dynamic tracking system 1 may be a plant. Using the dynamic tracking system 1, it is possible to monitor the growth and morphological changes of a plant, for example, under constant light conditions. If the algorithm of this dynamic tracking system 1 is generalized, it can also be applied to more detailed analysis of plants (e.g., measurement of growth patterns, shape changes, and reaction rates).

[0125] In summary, the movement tracking system 1 can be used for the following purposes: (1) Animal behavior analysis: for example, it can be used to analyze the movement of nematodes, mice, insects, zebrafish, etc. (2) Plant growth tracking: for example, it can be used to analyze the growth patterns of leaves and stems. (3) Pharmaceutical testing: for example, it can be used to observe pharmacological effects. (4) Ecological monitoring: for example, it can be used to record the migration routes and energy consumption of insects and birds.

[0126] The hardware and software configurations of the motion tracking system 1 are examples only and can be changed and modified as needed.

[0127] The core processing part of the motion tracking system 1, which consists of a CPU 31, main memory 32, external memory 33, operation input unit 34, display 35, communication interface 36, and internal bus 37, can be implemented using a normal computer system, not a dedicated system. For example, the motion tracking system 1 that performs the above processing may be configured by distributing a computer program for executing the above operations on a computer-readable recording medium (flexible disk, CD-ROM, DVD-ROM, etc.) and installing the computer program on a computer. Alternatively, the motion tracking system 1 may be configured by storing the computer program on a storage device of a server device on a communication network such as the Internet and downloading it using a normal computer system.

[0128] If the functions of the motion tracking system 1 are realized through a division of labor between the OS (operating system) and the application program, or through collaboration between the OS and the application program, then only the application program portion may be stored on the recording medium or storage device.

[0129] It is also possible to superimpose a computer program onto a carrier wave and distribute it via a communication network. For example, a computer program could be posted on a bulletin board system (BBS) on a communication network and distributed via the network. This computer program could then be launched and executed under the control of the OS, similar to other application programs, thereby enabling the aforementioned processing.

[0130] This invention allows for various embodiments and modifications without departing from the broad spirit and scope of the invention. Furthermore, the embodiments described above are for illustrative purposes only and do not limit the scope of the invention. In other words, the scope of the invention is indicated not by the embodiments, but by the claims. Various modifications made within the scope of the claims and the equivalent scope of the meaning of the invention are considered to be within the scope of the invention.

[0131] This application claims priority based on Japanese Patent Application No. 2025-4004, filed on January 10, 2024, and incorporates the entire description, claims, and drawings of Japanese Patent Application No. 2025-4004 by reference within this specification.

[0132] This invention is useful for evaluating the dynamics of living organisms.

[0133] 1. Motion tracking system, 2. Imaging device, 3. Data processing device, 4. Image processing unit, 5. Human-machine interface (MMI), 10. Image acquisition unit, 11. Region of interest identification unit, 12. Contour detection unit, 13. Motion feature calculation unit, 14. Storage unit, 15. Analysis unit, 16. Multithreaded control unit, 20. Graphical user interface (GUI), 21. Imaging adjustment unit, 22. Image adjustment unit, 23. Region of interest adjustment unit, 24. Video display unit, 25. Contour adjustment unit, 26. Motion feature adjustment unit, 27. Storage adjustment unit, 28. Analysis adjustment unit, 31. CPU, 32. Main memory, 33. External memory, 34. Operation input unit, 35. Display, 36. Communication interface (I / F), 37. Internal bus, 38. Program, C, C' Contour, CP Contour point, E Minimum circle, F Frame image, O Center position, R Region of interest, RI Image, RP Point, T1, T2 living organism

Claims

1. A motion tracking system comprising: a contour detection unit that detects the contour of a living organism in a frame image constituting a video and performs a process for each frame image to set multiple contour points aligned along the contour; a video display unit that displays a video composed of frame images in which the set multiple contour points are superimposed via a graphic user interface; and a contour adjustment unit that can adjust the positions of the multiple contour points in the displayed video via a graphic user interface.

2. The motion tracking system according to claim 1, comprising: a motion feature calculation unit that calculates motion features of a living organism based on the plurality of contour points; and a storage unit that stores tracking data including the frame image, the time the frame image was captured, data representing the contour of the living organism detected in the frame image, and data representing the motion features of the living organism calculated in the frame image.

3. The motion tracking system according to claim 2, comprising: an image acquisition unit that sequentially acquires frame images from an imaging device and corrects the frame images using image processing parameters; and a multithreaded control unit that controls the image acquisition unit, the contour detection unit, and the motion feature calculation unit so that processing is performed in parallel on a frame image basis.

4. The dynamic tracking system according to claim 2, further comprising an analysis unit that performs analysis processing on the tracking data stored in the memory unit to generate biological analysis data, and generates plot data that represents the analysis data in a predetermined number of dimensional spaces.

5. The motion tracking system according to claim 1, wherein the contour detection unit assigns identification information to each contour point, and the video display unit displays the contour points based on the identification information so that the corresponding contours can be identified.

6. The motion tracking system according to claim 5, wherein, when there are multiple overlapping contours in the frame image, the contour detection unit assigns identification information for each of the multiple overlapping contours to the contour points of the overlapping portion.

7. The motion tracking system according to claim 1, comprising a region of interest adjustment unit that adjusts a region of interest within the frame image via a graphic user interface, and the contour detection unit that detects the contour of a living organism within the region of interest.

8. The motion tracking system according to claim 7, wherein the region of interest adjustment unit is capable of adjusting the shape of the region of interest.

9. The motion tracking system according to claim 7, wherein the region of interest adjustment unit converts an image within the region of interest into an image of the living organism viewed in the direction normal to the plane on which the living organism is placed within the region of interest.

10. The motion tracking system according to claim 1, wherein the contour detection unit performs filtering or smoothing processing on the detected contour.

11. The motion tracking system according to claim 1, wherein the contour detection unit detects the plurality of contour points so as to be arranged at equal intervals along the contour.

12. The motion tracking system according to claim 1, further comprising an imaging adjustment unit that allows adjustment of imaging parameters of an imaging device for imaging a living organism via a graphic user interface during imaging.

13. The motion tracking system according to claim 3, further comprising an image adjustment unit that allows adjustment of the image processing parameters during video display via a graphic user interface.

14. The motion tracking system according to claim 1, wherein the contour adjustment unit can adjust the detection parameters used for detecting the contour of a living organism by the contour detection unit while the video is being displayed.

Images