Method for generating gait analysis model based on fisheye images, gait analysis device using same, and method thereof
The method involves using a CCTV installed in a hospital, and are not suitable for elderly patients, and are not suitable for elderly patients, and are not suitable for elderly patients.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- GEOMEXSOFT
- Filing Date
- 2024-11-29
- Publication Date
- 2026-06-04
Smart Images
Figure KR2024019332_04062026_PF_FP_ABST
Abstract
Description
Method for generating a gait analysis model based on fisheye images, a gait analysis device using the same, and the method thereof
[0001] The present invention relates to a method for generating a gait analysis model based on fisheye images, a gait analysis device using the same, and a method thereof; more specifically, it relates to a technology for analyzing gait types and gait states using fisheye image data as input.
[0002] Conventional gait analysis methods for diagnosing a pedestrian's health condition or pathological symptoms by analyzing gait include methods using wearable sensors that measure the position of joints when worn on the human body, methods of analyzing by installing multiple cameras in a dedicated analysis space, and methods of analyzing gait using depth cameras.
[0003] In addition, such conventional gait analysis is performed using a treadmill or in a large analysis space equipped with a camera.
[0004] However, treadmills are not suitable for the elderly at risk of falls, and in the latter case, there is a problem that analysis is difficult in confined spaces. Additionally, wearable sensors, multiple cameras, or depth cameras are expensive, and experts are required for analysis.
[0005] In order to solve the aforementioned problems, the present invention aims to provide a method for generating a gait analysis model based on fisheye images capable of analyzing gait types and gait states using fisheye image data from a CCTV installed in a hospital, etc. as input, a gait analysis device using the same, and the method.
[0006] The above objective is provided for a method for generating a gait analysis model for gait analysis based on fisheye images, comprising: a data collection step of collecting and synchronizing motion capture data and fisheye image data related to gait to construct a paired dataset; and a model learning step of performing machine learning using the paired dataset to construct a gait analysis model based on the fisheye image data. The above gait analysis model can be achieved by a method for generating a gait analysis model characterized by including: a first model trained to estimate the 3D joint coordinates of the pedestrian by extracting a feature map from the fisheye image data, taking the fisheye image data as input and the motion capture data as output; a second model trained to model the interrelationships between the joints and extract dependency information between the joints by taking the 3D joint coordinates of the pedestrian extracted from the first model as input; and a third model trained to classify the gait type of the pedestrian by combining the image feature map extracted from the first model and the dependency information between the joints output from the second model, and to analyze the gait state of the pedestrian by correcting the 3D joint coordinate data through analysis according to a time series.
[0007] In addition, the first model is an HRNetv2-based model and can integrate and process a first feature map that extracts detailed features from a high-resolution stream to provide 3D joint coordinates and posture information of detailed joints, a second feature map that extracts medium-scale features from a medium-resolution stream to provide information on relationships or movements between joints, a third feature map that extracts global context from a low-resolution stream to provide information on the flow of posture and body arrangement, and a fourth feature map that provides information on body structure or posture changes from an ultra-low-resolution stream.
[0008] In addition, the first model may include a first algorithm that assigns weights to each joint according to the importance of each joint in the walking motion of a pedestrian when learning fisheye image data, and a second algorithm that corrects the estimated 3D joint coordinates using a loss function.
[0009] In addition, the second model may include a Graph Convolutional Network (GCN) based model, and the third model may include a Long Short-Term Memory (LSTM) model.
[0010] And, the above gait type includes at least three of normal gait, painless gait, staggering gait, stair gait, spastic leg gait, Trendelenburg gait, cerebellar ataxia gait, sensory ataxia gait, Parkinson's gait, dystonia gait, freezing gait, and frontal lobe gait disorder, and the above gait state may include at least three of walking speed, average stride length, walking balance, posture sway, and average stance.
[0011] The above objective can also be achieved by a gait analysis device comprising: a storage unit in which a gait analysis model generated by learning a paired dataset constructed by synchronizing motion capture data and fisheye image data regarding walking is stored; and a processor that inputs the fisheye image data into the gait analysis model to analyze the walking type and walking state of a pedestrian included in the fisheye image data; wherein the gait analysis model comprises: a first model learned to estimate the 3D joint coordinates of the pedestrian by extracting a feature map from the fisheye image data, with the fisheye image data as input and the motion capture data as output; a second model learned to model the interrelationship between the joints and extract dependency information between the joints by inputting the 3D joint coordinates of the pedestrian extracted from the first model; and a third model learned to classify the walking type of the pedestrian by combining the image feature map extracted from the first model and the dependency information between the joints output from the second model, and to analyze the walking state of the pedestrian by correcting the 3D joint coordinate data through analysis according to a time series.
[0012] The above objective can also be achieved by a gait analysis method comprising: a step of estimating the 3D joint coordinates of the pedestrian by extracting a feature map from fisheye image data using a first model; a step of modeling the interrelationships between the joints and extracting dependency information between the joints using a second model with the 3D joint coordinates of the pedestrian extracted from the first model as input; and a step of classifying the gait type of the pedestrian by combining the feature map extracted from the first model and the dependency information between the joints output from the second model using a third model, and analyzing the gait state of the pedestrian by correcting the 3D joint coordinate data through time series analysis.
[0013] As described above, the method for generating a walking analysis model based on fisheye images according to the present invention, the walking analysis device using the same, and the method thereof can continuously analyze walking types, walking states, and trends using only CCTV fisheye image data without using complex equipment.
[0014] FIG. 1 is a conceptual diagram of a gait analysis system according to one embodiment of the present invention.
[0015] FIG. 2 is a flowchart relating to a method for generating a gait analysis model according to one embodiment of the present invention.
[0016] FIG. 3 schematically illustrates the structure of a gait analysis model according to one embodiment of the present invention.
[0017] FIG. 4a is an example of fisheye image data taken of a hospital hallway according to one embodiment of the present invention.
[0018] FIG. 4b is an example of an image visualizing OKS sigma according to one embodiment of the present invention.
[0019] FIG. 4c illustrates an example of 3D joint coordinate data extracted from a first model according to an embodiment of the present invention.
[0020] FIG. 4d illustrates a walking state analysis image analyzed according to one embodiment of the present invention.
[0021] FIG. 5 is a flowchart relating to a gait analysis method using a gait analysis model according to one embodiment of the present invention.
[0022] FIGS. 6a and 6b illustrate an example of a gait analysis result screen provided by a gait analysis system according to an embodiment of the present invention.
[0023] Hereinafter, specific embodiments of the present invention will be described with reference to the drawings.
[0024] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, detailed descriptions of known functions or configurations that may obscure the essence of the present invention in the following description and the accompanying drawings are omitted. Additionally, it should be noted that identical components throughout the drawings are indicated by the same reference numerals whenever possible.
[0025] The terms and words used in the specification and claims described below should not be interpreted as being limited to their ordinary or dictionary meanings, but should be interpreted in a meaning and concept consistent with the technical spirit of the invention, based on the principle that the inventor can appropriately define the concept of the terms to best describe his invention. Accordingly, the embodiments described in this specification and the configurations illustrated in the drawings are merely the most preferred embodiments of the invention and do not represent all of the technical spirit of the invention; therefore, it should be understood that various equivalents and modifications that can replace them may exist at the time of filing this application.
[0026] FIG. 1 is a conceptual diagram of a gait analysis system according to an embodiment of the present invention. A gait analysis system according to an embodiment of the present invention may include a gait analysis device (100), a camera (10) connected thereto via a network (20), a management system (200), and a user terminal (210).
[0027] The camera (10) is intended to acquire images of the surveillance area and, for example, may be a CCTV fisheye camera (10A~10N) installed on the ceiling of the surveillance area. Fisheye cameras (10A~10N) are very economical because they can capture the entire corridor or room when installed in an indoor corridor or room, and since they are 2D images, they have the advantage of allowing real-time analysis with less burden on data analysis. The camera (10) transmits the captured images to a gait analysis device (100). The camera (10) and the gait analysis device (100) are connected via a wired or wireless network. In the first embodiment of the present invention, the camera (10) is installed in the corridor and patient room of a hospital ward, and images acquired through the camera (10) are transmitted to the gait analysis device (100) as an example.
[0028] The walking analysis device (100) is designed to analyze the walking type and walking state of a pedestrian by analyzing images received from a camera (10), and can be implemented as at least one computing device and computing system including a processor (110) and a storage unit (220) in which at least one program is stored.
[0029] The processor (110) is intended to execute and process computer program instructions by performing basic logic, calculations, operations, etc., and computer program code stored in the storage unit (120) is loaded into the processor (110) and executed. The processor (110) executes the algorithm stored in the storage unit (120) to learn a walking analysis model (300) for extracting the walking type and walking state of a pedestrian from fisheye image data, and can perform walking analysis using the learned walking analysis model (300).
[0030] The storage unit (120) may be a computer-readable recording medium and may include a permanent mass storage device such as RAM (random access memory), ROM (read only memory), and a disk drive. Additionally, at least one program code executed by the processor (110) may be stored in the storage unit (120). Such program code may be loaded into the storage unit (120) from a separate floppy drive, disk, tape, DVD / CD-ROM drive, memory card, etc. The storage unit (120) may include a gait analysis model (300) for analyzing gait types and gait states based on fisheye image data, an algorithm for learning the gait analysis model (300), and an algorithm for executing the gait analysis model (300). Additionally, the storage unit (120) may further include an algorithm for detecting dangerous situations based on gait analysis results, or an algorithm for identifying and tracking patients.
[0031] The gait analysis device (100) may include a DB (130), a communication interface (150), and a user interface (140). The DB (130) includes, for example, a patient DB, a gait analysis DB for each patient, and various DBs necessary for safety management. The communication interface (150) is for communicating with other devices through a network (20) and may include one or more arbitrary networks (20) among, for example, a PAN (personal area network), LAN (local area network), CAN (campus area network), MAN (metropolitan area network), WAN (wide area network), BBN (broadband network), and the Internet. The present invention may be configured as a closed network rather than an open network for the protection of personal information. Meanwhile, the user interface (140) may include, for example, an input device such as a microphone, a keyboard, or a mouse, and an output device such as a display and a speaker.
[0032] The gait analysis device (100) can store the results of a patient's gait analysis analyzed through a gait analysis model (300) in a DB and periodically generate a report to provide to a management system (200) and / or a user terminal (210).
[0033] The management system (200) is a system for managing the patient's condition and can display the gait analysis results transmitted from the gait analysis device (100) or access the gait analysis device (100) to check the patient's gait analysis results. The management system can be implemented as a computing system linked with the gait analysis device (100). The user terminal (210) may include a smartphone, tablet PC, etc., which is linked with the gait analysis device (100) and can check the gait analysis results transmitted from the gait analysis device (100).
[0034] The walking analysis device (100) includes a walking analysis model (300) for identifying objects in fisheye image data and estimating the posture of a pedestrian to analyze walking types and walking states, and the walking analysis model (300) is stored in a storage unit (120) and executed by a processor (110).
[0035] Meanwhile, the gait analysis device (100) may further include a visualization unit that visualizes and displays the gait analysis results.
[0036] Hereinafter, a walking analysis model (300) according to one embodiment of the present invention will be described with reference to FIGS. 2 to 4d.
[0037] FIG. 2 is a flowchart relating to a method for generating a gait analysis model (300) according to one embodiment of the present invention, and FIG. 3 schematically illustrates the structure of a gait analysis model (300) according to one embodiment of the present invention.
[0038] Referring to FIG. 2, for model training, the device (100) collects and synchronizes motion capture data and fisheye image data regarding walking to construct a paired dataset (S10). Motion capture data can be obtained through a wearable sensor (e.g., a motion capture suit) that specifies the joint positions of a person, and a motion capture system that receives signals from the sensor to acquire the joint positions. A person wears a motion capture suit and performs various walking types or walking movements according to a predefined scenario to acquire joint coordinate data through the motion capture system, while simultaneously acquiring fisheye image data through a CCTV installed on the ceiling. The joint positions for walking analysis may include joints of the lower body such as the feet, foreskin, heels, knees, and hips, as well as joints of the upper body such as the head, shoulders, elbows, or hands.
[0039] Motion capture data and fisheye image data are synchronized and stored as a paired dataset, which enables learning the accurate joint position in each frame of the fisheye image data.
[0040] Video data from a fisheye CCTV using a fisheye lens has little distortion in the center of the screen, but severe non-linear distortion occurs as it moves toward the periphery, resulting in a difference from the actual body proportions. Figure 4a is an example of fisheye video data taken of a hospital hallway according to an embodiment of the present invention, and it can be seen that severe distortion occurs in the periphery of the image.
[0041] A gait analysis model (300) according to one embodiment of the present invention can correct distortion and estimate accurate joint positions using only fisheye image data by learning the distorted coordinates of motion capture data and fisheye image data. By training the model through the paired dataset constructed in this way, motion capture data can be reproduced using only fisheye lens images without complex equipment, thereby enabling accurate analysis of a patient's gait.
[0042] The device (100) augments the learning data obtained through the aforementioned process by rotating it at an arbitrary angle (S11). Since fisheye image data captured with a fisheye lens maintains the same type of distortion even when rotated at various angles, rotational augmentation can be used to achieve the effect of adding various new data. This augmentation technique enables the recognition and response to changes in posture that may occur in various directions and angles.
[0043] Additionally, the device (100) can perform augmentation by methods such as left-right inversion of fisheye image data, brightness adjustment, image position adjustment, and addition of random noise, and through such various augmentation techniques, the walking analysis model (300) can maintain robust performance even under various environmental conditions and lighting changes.
[0044] The device (100) constructs a gait analysis model (300) by performing machine learning using a pair data set and training data generated by rotation, etc. (S13).
[0045] Referring to FIG. 3, a gait analysis model (300) according to one embodiment of the present invention is a deep learning-based model and includes a first model (310), a second model (320), and a third model (330).
[0046] For gait analysis, data from the start to the end of a pedestrian's gait must be analyzed in a time series, and for this purpose, the gait analysis model (300) uses a structure in which a first model (310), a second model (320), and a third model (330) are combined.
[0047] The first model (310) is a model trained to estimate the 3D joint coordinates of the pedestrian by extracting a feature map from fisheye image data, and may be a Convolutional Neural Network (CNN) based model, for example, an HRNetv2 based model. The first model (310) according to the present invention can estimate 3D joint coordinate data from severely distorted fisheye image data by training the model using a pair dataset of synchronized 2D fisheye image data and 3D motion capture data.
[0048] A first model according to one embodiment of the present invention is a deep learning model that extracts features from high resolution to low resolution by parallel processing of multi-stage feature maps, and has a structure that optimizes pose estimation performance by maintaining high-resolution features and processing multiple resolutions in parallel. The first model (310) maintains high-resolution feature maps from the beginning of the network while integrating information obtained by processing feature maps of various resolutions in parallel, and thus, feature maps of each resolution are combined complementarily to form more precise feature representations as the network becomes deeper. Specifically, the first model (310) can process information of various resolutions in parallel through four streams, such as a high-resolution stream, an intermediate-resolution stream, a low-resolution stream, and an ultra-low-resolution stream.
[0049] For example, the first model (310) captures detailed features from a high-resolution stream to accurately estimate small joint points or fine posture information, extracts medium-scale features from a medium-resolution stream, and extracts relationships and movements between major joints over a wider range. Additionally, the first model (310) captures global context from a low-resolution stream to extract information useful for understanding the flow of the posture or the overall arrangement of the body, and captures abstract features from an ultra-low-resolution stream to extract body structure and posture changes in a broader sense.
[0050] The first model (310) according to the present invention is capable of capturing features of different resolutions, thereby harmoniously combining various information ranging from detailed information of small joints to the overall position of the whole body. In this way, the first model (310) according to the present invention can achieve very high performance in gait analysis by simultaneously utilizing multi-resolution feature maps.
[0051] Meanwhile, the first model (310) may use a first algorithm (e.g., OKS (Object Keypoint Similarity) Sigma) that assigns weights to each joint according to the importance of each joint in the walking motion of a pedestrian when learning fisheye image data. In the joint coordinate estimation task for walking analysis, the first algorithm may assign different weights to joints of major parts of the human body according to observability or importance of location. For example, high weights may be assigned to joints of relatively high importance, such as the shoulder or ankle, which are used in walking analysis, and low weights may be assigned to joints of low importance, such as the head or wrist. FIG. 4b is an example of an image visualizing the OKS Sigma according to an embodiment of the present invention. Referring to FIG. 4b, the joints to which high weights are assigned are shown as large and red.
[0052] The first model (310) may include a second algorithm that corrects the estimated 3D joint coordinates using a loss function. The second algorithm may be a Joints Mean Squared Error (MSE) loss function expressed by Equation 1, which minimizes the error occurring in joint position prediction by squaring the difference between the predicted joint coordinates and the actual joint coordinates and then calculating the average. This allows the first model (310) to be guided to predict the coordinates of each joint closer to the actual position during training.
[0053]
[0054] Here, N is the total number of joints, is the coordinates of joint i predicted by the model, y i represents the actual coordinates of joint i. Therefore, by applying OKS sigma to the joint MSE loss function, a loss function (L) containing weights reflecting the importance of each joint is obtained. pose ) is equal to mathematical formula 2.
[0055]
[0056] In this way, the first model (310) according to the present invention can learn by applying a loss function that reflects OKS sigma, thereby increasing the positional accuracy of each joint while also considering the importance of each joint.
[0057] FIG. 4c illustrates an example of 3D joint coordinate data extracted from a first model (310) according to an embodiment of the present invention. The 3D joint coordinate data extracted from the first model (310) is input into a second model (320) to be described later. Additionally, the first model (310) uses a feature map extracted from the last layer of a model (e.g., HRNetv2) to capture features of an image frame, and this feature map is input into a third model (330) to be described later and utilized as visual information.
[0058] The second model (320) is a model that learns to model the interrelationships between joints and extract dependency information between joints using the 3D joint coordinates of the pedestrian extracted from the first model (310) as input, and may be a GCN-based model. The second model (320) learns the structural relationships between joints using the 3D joint coordinate data extracted from the first model (310) as input. Through this, it estimates the connection relationships and interdependencies between joints. The second model (320) receives 3D joint coordinate data as input and models the interrelationships between each joint. Joints are represented as nodes of a graph, and the connection relationships between joints are represented as edges, allowing the model to learn the dependencies between joints. Through this, the model can recognize not only the time-series data of joint coordinates but also the structural relationships between joints. For example, the GCN-based second model (320) can convert the joint coordinates into a graph structure to receive the position and relationship of each joint in the form of a graph, and can extract the dependency information between joints learned through GCN and transmit it to the third model (330).
[0059] The third model (330) is trained to classify the walking type of a pedestrian by combining the image feature map extracted from the first model (310) and the structural dependency (i.e., interrelationship) information between joints output from the second model (320), and is intended to analyze the walking state by correcting 3D joint coordinate data through analysis based on time series, and may include an LSTM model. Through this, the third model (330) can learn all information reflecting the structural relationship of the joints along with the time series data.
[0060] Meanwhile, the third model (330) may include a function to correct joint coordinates based on time series data, thereby increasing the accuracy of the joint coordinates extracted from the first model (310), which is a pose estimation model.
[0061] The third model (330) takes the outputs of the first model (310) and the second model (320) as inputs and classifies and outputs a gait type. Here, the gait type may include at least three of normal gait, antalgic gait, lurching gait, steppage gait, stiff-legged gait, Trendelenburg gait, crerbellar ataxic gait, sensory ataxic gait, Parkinsonian gait, dystonic gait, gait freezing, and frontal gait disorder. The third model (330) is trained to classify a gait type into any one of the above gait types based on fisheye image data.
[0062] The third model (330) takes the outputs of the first model (310) and the second model (320) as inputs to analyze and output a walking state. The analysis of the walking state plays an important role in evaluating the patient's mobility and balance state to accurately determine physical health and recovery status. In the present invention, the walking state subject to analysis may include at least three of walking speed, average stride length, walking balance, posture sway, and average stance.
[0063] FIG. 4d illustrates a walking state analysis image analyzed according to an embodiment of the present invention. Referring to FIG. 4d, a walking analysis model (300) analyzes fisheye image data in a time series to extract walking state values such as stride length, walking speed, balance, posture sway, and stance.
[0064] Stride length is an important indicator of a pedestrian's stride size, and to measure this, the third model (330) estimates the actual distance between the two ankles through the distance between the left and right ankle keypoints extracted from the first model (310) and the second model (320). The third model (330) can calculate the average stride length and can record changes in stride length using the maximum stride length value in the recent frame, taking into account the patient's gait cycle.
[0065] Walking speed is an important factor in evaluating a pedestrian's mobility ability, and the third model (330) calculates the distance traveled by tracking the center point of the hips, that is, the midpoint between the left and right hip key points, to calculate the speed. Since the location of this center point of the hips accurately reflects the movement of the body's center of gravity during walking, the speed can be calculated based on this. The third model (330) calculates the distance traveled by the center of the hips between consecutive frames and then divides this by time to calculate the walking speed. Through this, it is possible to analyze how consistently the body's movement is maintained and how much mobility ability improves or decreases.
[0066] Postural sway is an indicator that measures body balance during walking and is a factor that can evaluate how stable the pedestrian's center of gravity is. The third model (330) establishes a spinal centerline by connecting the shoulder center point and the hip center point, which are known to play an important role in maintaining body balance during walking, and can measure the direction and degree of sway through changes in the inclination of this spinal centerline.
[0067] The third model (330) analyzes the sway of the posture by calculating the angle formed by the center line of the spine and the vertical line, with positive values indicating a tilted state to the left and negative values indicating a tilted state to the right. Through this, the left-right sway of the body occurring during walking can be quantitatively evaluated, which can identify the patient's ability to maintain balance or unstable walking patterns.
[0068] Average stance refers to the distance between the two feet during walking, i.e., the stride width, and is an important factor in evaluating a stable gait. The third model (330) measures the distance between the two feet by finding the point where the distance between the two feet is minimized during the walking process, and through this, can evaluate how much body stability is maintained during walking. The narrower the stance, the more difficult it may be to maintain body balance.
[0069] Referring again to FIG. 2, the device (100) evaluates and verifies the performance of the gait analysis model (300) built by learning using test data and adjusts the learning data and model (S15).
[0070] FIG. 5 is a flowchart relating to a walking analysis method using a walking analysis model (300) according to one embodiment of the present invention.
[0071] Referring to FIG. 5, the device (100) uses the first model (310) to extract a feature map from input fisheye image data and estimates and outputs the 3D joint coordinates of the pedestrian (S20).
[0072] The device (100) uses two models to model the interrelationships between joints and extracts dependency information between joints using the 3D joint coordinates of a pedestrian extracted from the first model (310) as input (S21).
[0073] The device (100) uses the third model (330) to combine the feature map extracted from the first model (310) and the 3D joint coordinate data output from the second model (320) to classify walking types and analyze walking states (S23).
[0074] The device (100) visualizes the results of the analysis of walking types and walking states classified through the walking analysis model (300) through a visualization unit and displays them on a display (S25).
[0075] FIGS. 6a and 6b illustrate an example of a gait analysis result screen provided by a gait analysis system according to an embodiment of the present invention.
[0076] Referring to FIG. 6a, the left side displays an image of footprints and postures extracted from fisheye image data, and the right side displays a graph of left-right balance, walking speed, and left-right stride length calculated based on parameter values extracted from the gait analysis model (300). By expressing the results analyzed by the gait analysis model (300) as a graph, the visualization unit can visually analyze the patient's walking rhythm, changes in stride size, left-right balance, and walking speed.
[0077] Referring to Fig. 6b, the joint data extracted from the image and changes in stride length are shown on the left, and the average stride length, walking speed, knee flexion angle, and walking balance values are displayed on the right, and time-series data is displayed as a graph, allowing for intuitive verification of walking patterns, trends of change, and changes in pose.
[0078] In this way, the walking analysis device (100) and the method according to the present invention can accurately analyze the walking of a pedestrian using only 2D fisheye image data without complex equipment.
[0079] Meanwhile, a gait analysis model (300) according to one embodiment of the present invention may further include algorithms for identifying and tracking people. For example, the DB may include personal information and identification information of patients, and may include a gallery section where images of patients are stored. Meanwhile, each bed in the facility is assigned a bed identification number, and the DB of the gait analysis device (100) has registered the room number, location information, and bed identification number where each bed is located. The gallery section stores the bed identification number corresponding to the bed assigned to the patient, mapped to the image of the target object. Since the present invention (re)identifies the patient based on whether or not there is interaction with the bed, the bed identification number may be assigned as the patient identification number. For example, a bed identification number may be assigned to each patient together with or in place of the patient identification number, and the patient may be identified according to the bed identification number.
[0080] Conventionally, human identification is generally performed by comparing facial feature points. However, since the face is not captured when using fisheye image data as in the present invention, such conventional technology is not suitable for the present invention; furthermore, the patient image registered in the gallery section is an image of the patient viewed from above (ceiling), extracted from the fisheye image, instead of a photograph of the face.
[0081] If it is detected that the first patient is lying or sitting on the first bed, the gait analysis device (100) identifies the first target object detected in the image as the target object assigned to the first bed, and the first target object is assigned bed identification information (or patient ID), so it can be continuously tracked in subsequent images and identified as the first target object (patient).
[0082] When initially registering a patient's image in the gallery, the patient ID can be identified based on whether there is interaction with the bed as described above, and the image can be registered in the gallery. Subsequently, even if tracking is interrupted or accurate identification is difficult due to various circumstances, the patient can be accurately identified when the patient returns to the bed and lies down or sits.
[0083] Through this patient identification and tracking technology, gait analysis results can be stored for each patient, and the patient's gait status can be verified.
[0084] In the aforementioned embodiments, the analysis of fisheye image data of a hospital corridor or patient room was described as one example, but it should be noted that the present invention is not limited thereto.
[0085] The present invention may also be implemented on various computer-readable recording media, such as magnetic storage media, optical reading media, and digital storage media, in which a computer program for performing the method according to the present invention is stored when executed on a computer. Furthermore, the descriptions expressed as steps in the claims are not limited to the order of steps.
[0086] Although several embodiments of the present invention have been described so far, those skilled in the art will understand that it is possible to modify or substitute parts of the embodiments without departing from the technical spirit of the present invention. Therefore, the scope of protection of the present invention should be considered to extend to the invention described in the claims and its equivalents.
Claims
1. In a method for generating a gait analysis model for gait analysis based on fisheye images, A data collection step for constructing a paired dataset by collecting and synchronizing motion capture data and fisheye image data regarding walking; and It includes a model learning step for constructing a gait analysis model based on the fisheye image data by performing machine learning using the above pair dataset; The above gait analysis model is, A first model trained to take the above fisheye image data as input and the above motion capture data as output, extract a feature map from the above fisheye image data, and estimate the 3D joint coordinates of the pedestrian, A second model trained to model the interrelationships between the joints and extract dependency information between the joints using the 3D joint coordinates of the pedestrian extracted from the first model as input, and A method for generating a gait analysis model, characterized by including a third model that is trained to classify the gait type of the pedestrian by combining an image feature map extracted from the first model and dependency information between the joints output from the second model, and analyzes the gait state of the pedestrian by correcting the 3D joint coordinate data through analysis based on a time series.
2. In Paragraph 1, A gait analysis device characterized by the above-mentioned first model being an HRNetv2-based model, which integrates and processes a first feature map that extracts detailed features from a high-resolution stream to provide 3D joint coordinates and posture information of detailed joints, a second feature map that extracts medium-scale features from a medium-resolution stream to provide information on relationships or movements between joints, a third feature map that extracts global context from a low-resolution stream to provide information on the flow of posture and body arrangement, and a fourth feature map that provides information on body structure or posture changes from an ultra-low-resolution stream.
3. In Paragraph 1, A walking analysis device characterized by including the first model, a first algorithm that assigns weights to each joint according to the importance of each joint in the walking motion of a pedestrian when learning fisheye image data, and a second algorithm that corrects the estimated 3D joint coordinates using a loss function.
4. In Paragraph 1, The above second model includes a GCN-based model, and A gait analysis device characterized by the above-mentioned third model including an LSTM model.
5. In Paragraph 1, The above gait types include at least three of normal gait, painless gait, staggering gait, stair gait, spastic leg gait, Trendelenburg gait, cerebellar ataxic gait, sensory ataxic gait, Parkinsonian gait, dystonic gait, freezing gait, and frontal lobe gait disorder, and A walking analysis device characterized in that the above walking state includes at least three of walking speed, average stride length, walking balance, posture sway, and average stance.
6. In a gait analysis device, A storage unit storing a gait analysis model generated by training a paired dataset constructed by synchronizing motion capture data and fisheye image data regarding gait; and It includes a processor that inputs the fisheye image data into the gait analysis model to analyze the gait type and gait state of a pedestrian included in the fisheye image data; The above gait analysis model is, A first model trained to take the above fisheye image data as input and the above motion capture data as output, extract a feature map from the above fisheye image data, and estimate the 3D joint coordinates of the pedestrian, A second model trained to model the interrelationships between the joints and extract dependency information between the joints using the 3D joint coordinates of the pedestrian extracted from the first model as input, and A gait analysis device characterized by including a third model that is trained to classify the gait type of the pedestrian by combining an image feature map extracted from the first model and dependency information between the joints output from the second model, and analyzes the gait state of the pedestrian by correcting the 3D joint coordinate data through analysis based on a time series.
7. In Paragraph 6, A gait analysis device characterized by the above-mentioned first model being an HRNetv2-based model, which integrates and processes a first feature map that extracts detailed features from a high-resolution stream to provide 3D joint coordinates and posture information of detailed joints, a second feature map that extracts medium-scale features from a medium-resolution stream to provide information on relationships or movements between joints, a third feature map that extracts global context from a low-resolution stream to provide information on the flow of posture and body arrangement, and a fourth feature map that provides information on body structure or posture changes from an ultra-low-resolution stream.
8. In Paragraph 7, A walking analysis device characterized by including the first model, a first algorithm that assigns weights to each joint according to the importance of each joint in the walking motion of a pedestrian when learning fisheye image data, and a second algorithm that corrects the estimated 3D joint coordinates using a loss function.
9. In Paragraph 6, The above second model includes a GCN-based model, and A gait analysis device characterized by the above-mentioned third model including an LSTM model.
10. In Paragraph 6, The above gait types include at least three of normal gait, painless gait, staggering gait, stair gait, spastic leg gait, Trendelenburg gait, cerebellar ataxic gait, sensory ataxic gait, Parkinsonian gait, dystonic gait, freezing gait, and frontal lobe gait disorder, and A walking analysis device characterized in that the above walking state includes at least three of walking speed, average stride length, walking balance, posture sway, and average stance.
11. Regarding gait analysis methods, A step of estimating the 3D joint coordinates of the pedestrian by extracting a feature map from fisheye image data using a first model, A step of using a second model to model the interrelationships between the joints and extract dependency information between the joints, using the 3D joint coordinates of the pedestrian extracted from the first model as input; and A walking analysis method that classifies the walking type of the pedestrian by combining the feature map extracted from the first model and the dependency information between the joints output from the second model using a third model, and analyzes the walking state of the pedestrian by correcting the 3D joint coordinate data through analysis based on a time series.
12. In Paragraph 11, A gait analysis method characterized by performing learning using a paired dataset constructed by synchronizing motion capture data and fisheye image data regarding gait in the first model above.
13. In Paragraph 11, A gait analysis method characterized by integrating and processing a first feature map, which is an HRNetv2-based model, that extracts detailed features from a high-resolution stream to provide 3D joint coordinates and posture information of detailed joints; a second feature map, which extracts medium-scale features from a medium-resolution stream to provide information on relationships or movements between joints; a third feature map, which extracts global context from a low-resolution stream to provide information on the flow of posture and body arrangement; and a fourth feature map, which provides information on body structure or posture changes from an ultra-low-resolution stream.
14. In Paragraph 13, A walking analysis method characterized by including a first algorithm that assigns weights to each joint according to the importance of each joint in the walking motion of a pedestrian when learning fisheye image data, and a second algorithm that corrects the estimated 3D joint coordinates using a loss function.
15. In Paragraph 11, The above second model includes a GCN-based model, and A gait analysis method characterized in that the above-described third model includes an LSTM model.
16. In Paragraph 11, The above gait types include at least three of normal gait, painless gait, staggering gait, stair gait, spastic leg gait, Trendelenburg gait, cerebellar ataxic gait, sensory ataxic gait, Parkinsonian gait, dystonic gait, freezing gait, and frontal lobe gait disorder, and A walking analysis method characterized in that the above walking state includes at least three of walking speed, average stride length, walking balance, posture sway, and average stance.
17. In Paragraph 11, A pedestrian analysis method characterized by further including the step of identifying and tracking a monitoring target pedestrian in the above fisheye image data.