Feature point tracking device and program

Abstract

One aspect of the present invention is provided with: a first processing unit that determines the success or failure of detection of a feature point in a first image frame among a plurality of image frames that are included in a moving image and are successive in time series; a second processing unit that determines the success or failure of detection of the feature point in a second image frame positioned before the first image frame in time series when it is determined that the detection of the feature point in the first image frame has failed; a third processing unit that sets a parameter for a template on the basis of the detected position of the feature point detected from the second image frame when it is determined that the detection of the feature point in the second image frame has been successful; and a fourth processing unit that tracks the feature point in the first image frame by using the template represented by the parameter that has been set.

Description

Feature point tracking device and program

[0001] One aspect of this invention relates to a feature point tracking device and program used, for example, to track feature points of a target based on a moving image.

[0002] For example, in fields such as monitoring people and objects, techniques that detect characteristic points of the monitored target, such as people or objects, from video footage and track these characteristic points over time are widely used. A typical example of this is the technique that uses template matching.

[0003] Template matching works by pre-storing an image region containing the object to be tracked as a template image, and detecting the region of the object to be tracked from image frames input in a time series by detecting regions with a high similarity to the template image. However, template matching has the problem that the accuracy and stability of object tracking decrease due to changes in the image pattern representing the region of the object to be tracked caused by object deformation, occlusion by other objects, fluctuations in lighting, and noise in the sensor system.

[0004] Therefore, conventionally, template matching has been proposed that has a function to sequentially update the template according to, for example, the amount of movement of an object or the similarity between the object being tracked and the template (see, for example, Patent Document 1).

[0005] Japanese Patent No. 4070558

[0006] However, in the template matching method described in Patent Document 1, where the template is updated while performing template matching, if the object to be tracked crosses a background with many features, such as many edges or a different color from other areas, the template may deviate from the object to be tracked and be updated to the background with many features, making it difficult to track subsequent feature points.

[0007] This invention was made in view of the above circumstances, and aims to provide a technology that enables stable tracking of feature points by allowing feature point tracking even when the template parameters are changed due to events other than the target of tracking.

[0008] To solve the above problems, one embodiment of the feature point tracking device according to the present invention comprises: a first processing unit that determines whether or not feature point detection has been successful in a first image frame among a plurality of image frames that are consecutive in time series and included in a moving image; a second processing unit that determines whether or not feature point detection has been successful in a second image frame located earlier in time series than the first image frame if it is determined that feature point detection has failed in the first image frame; a third processing unit that sets template parameters based on the detection position of the feature point detected from the second image frame if it is determined that feature point detection has been successful in the second image frame; and a fourth processing unit that tracks the feature point in the first image frame using the template represented by the set parameters.

[0009] According to one aspect of this invention, for example, even if a feature point becomes undetectable in a certain image frame of a moving image due to changes in the image or the influence of the background, the template parameters are set based on the detection position in a previous image frame where the feature point was detected. As a result, it becomes possible to track the feature point using the template defined by the set parameters.

[0010] In other words, according to one aspect of this invention, feature point tracking can be performed even if the template parameters are changed due to events other than the target of tracking, thereby providing a technology that enables stable tracking of feature points.

[0011] FIG. 1 is a diagram showing an example of a surveillance system including a feature point tracking device according to a first embodiment of the present invention. FIG. 2 is a block diagram showing an example of a hardware configuration of the feature point tracking device according to the first embodiment of the present invention. FIG. 3 is a block diagram showing an example of a software configuration of the feature point tracking device according to the first embodiment of the present invention. FIG. 4 is a flowchart showing an example of the procedure and processing content of feature point detection and tracking processing executed by the control unit of the feature point tracking device shown in FIG. 3. FIG. 5 is a flowchart showing an example of the procedure and processing content of the feature point detection processing among the processing procedures shown in FIG. 4. FIG. 6 is a flowchart showing an example of the procedure and processing content of the optimum parameter setting processing among the processing procedures shown in FIG. 4. FIG. 7 is a flowchart showing an example of the procedure and processing content of the search range setting processing among the processing procedures shown in FIG. 4. FIG. 8 is a diagram showing an example of feature points detected by the feature point detection processing shown in FIG. 5. FIG. 9A is a diagram for explaining an operation example of the optimum parameter setting processing shown in FIG. 6. FIG. 9B is a diagram for explaining an operation example of the optimum parameter setting processing shown in FIG. 6. FIG. 9C is a diagram for explaining an operation example of the optimum parameter setting processing shown in FIG. 6. FIG. 10 is a diagram showing an example of types of parameters used in the optimum parameter setting processing shown in FIG. 6 and their errors. FIG. 11 is a diagram showing an example of adjacency relationships of feature points used in the search range setting processing shown in FIG. 7.

[0012] Hereinafter, embodiments related to the present invention will be described with reference to the drawings.

[0013] [First Embodiment] (Configuration Example) (1) System FIG. 1 is a diagram showing an example of a surveillance system including a feature point tracking device according to the first embodiment of the present invention.

[0014] The surveillance system according to the first embodiment includes a camera CM arranged at a position capable of imaging a user US who is a surveillance target, and the camera CM is made connectable to the feature point tracking device CS via a network NW. In addition, a management terminal TM used by an administrator is made connectable to the feature point tracking device CS.

[0015] The camera CM is, for example, a network camera, which captures the whole body of the user US to be monitored and outputs moving image data thereof. Note that the camera CM is not limited to capturing the whole body of the user US, and may capture the half body, or may capture only a specific part such as an upper limb, a lower limb, or a face. Further, the camera CM may output a color image, may output a monochrome image, or an infrared camera may be used.

[0016] The management terminal TM is, for example, a general-purpose personal computer, and is used by an administrator to input various control instructions to the feature point tracking device CS, and to receive data indicating the detection and tracking results of feature points output from the feature point tracking device CS. Note that the management terminal TM is not limited to a personal computer, and may be a portable terminal such as a smartphone or a tablet terminal.

[0017] The network NW includes, for example, a wide area network centered on the Internet, and an access network for accessing the wide area network. As the access network, for example, a public data communication network adopting the 4G, 5G or 6G standard, or a LAN (Local Area Network) adopting the Wi-Fi (registered trademark) standard is used, but the access network is not limited thereto.

[0018] (2) Feature point tracking device CS Figure 2 and Figure 3 are block diagrams showing an example of the hardware configuration and software configuration of the feature point tracking device CS according to the first embodiment of the present invention.

[0019] The feature point tracking device CS is configured by, for example, a server computer disposed on the Web or on the cloud. Note that the feature point tracking device CS may be, for example, a personal computer used by an administrator.

[0020] The feature point tracking device CS includes a control unit 1 that uses a hardware processor such as a Central Processing Unit (CPU). A storage unit having a program storage unit 2 and a data storage unit 3, a sensor interface (hereinafter referred to as I / F) unit 4, and an input / output I / F unit 5 are connected to this control unit 1 via a bus 6.

[0021] The camera CM is connected to the sensor I / F unit 4. The sensor I / F unit 4 receives motion image data transmitted from the camera CM via the network NW.

[0022] A management terminal TM is connected to the input / output interface (I / F) unit 5. The I / F unit 5 transmits and receives various types of data with the management terminal TM. An external storage medium, such as a USB memory stick, may also be connected to the I / F unit 5.

[0023] The program storage unit 2, for example, combines a non-volatile memory that can be written to and read at any time, such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), with a non-volatile memory such as ROM (Read Only Memory), and stores application programs necessary to execute various processes according to the first embodiment of this invention, in addition to middleware such as an OS (Operating System).

[0024] The data storage unit 3 is, for example, a combination of non-volatile memory such as an HDD or SSD that can be written to and read at any time, and volatile memory such as RAM (Random Access Memory), and its storage area is provided with a video storage unit 31, a feature point detection model storage unit 32, a feature point storage unit 33, a parameter candidate storage unit 34, and a parameter storage unit 35.

[0025] The video storage unit 31 stores the video data transmitted from the camera CM for feature point detection and tracking processing.

[0026] The feature point detection model storage unit 32 stores data relating to the feature point detection model used to detect feature points from each image frame constituting the video data. The feature point detection model is, for example, a machine learning model such as a convolutional neural network (CNN) that has been pre-trained, and has the function of detecting the position of joints from a person's image and further detecting links that indicate the relationship between joints. As a feature point detection model, for example, "Openpose" developed by Carnegie Mellon University is preferred. "Openpose" is introduced in detail on the following website, so an explanation is omitted here.

[0027] Explanation of "Openpose": <URL: https: / / www.binarydevelop.com / article / pythonopencvopenpose-20708>.

[0028] The feature point storage unit 33 stores data representing the feature points of the monitored user US detected by the feature point detection model, in association with the identification information of the source image frame (hereinafter also referred to as the frame ID).

[0029] The parameter candidate storage unit 34 stores information representing candidate templates used when tracking the characteristic points of the monitored user US based on video data. Specifically, it stores multiple parameters that define the template. The parameters are defined, for example, by the width and height of the template and the width and height of the search range.

[0030] The parameter storage unit 35 stores the parameters of a template set for tracking the characteristic points of the monitored user US.

[0031] The control unit 1, as a processing unit according to the first embodiment of this invention, includes a video acquisition processing unit 11, a feature point detection processing unit 12, a feature point detection success / failure determination processing unit 13, an optimal parameter setting processing unit 14, a search range setting processing unit 15, and a feature point tracking processing unit 16.

[0032] Each of the above-mentioned processing units 11 to 16 is implemented by having the hardware processor of the control unit 1 execute an application program stored in the program storage unit 2. Note that some or all of the above-mentioned processing units 11 to 16 may be implemented using hardware such as LSI (Large Scale Integration) or ASIC (Application Specific Integrated Circuit).

[0033] During the period in which the video image acquisition processing unit 11 is detecting and tracking the characteristic points of the monitored user US, it receives video image data transmitted from the camera CM via the sensor I / F unit 4 and stores the received video image data along with information representing the time of capture in the video image storage unit 31.

[0034] The feature point detection processing unit 12 reads the video data stored in the video data storage unit 31 frame by frame, inputs the image data of the read image frame into the feature point detection model stored in the feature point detection model storage unit 32, and obtains feature point detection data from the feature point detection model. The feature point detection processing unit 12 then stores the obtained feature point detection data in the feature point storage unit 33, associating it with the frame ID of the source image frame.

[0035] The feature point detection success / failure determination processing unit 13 determines, for each image frame, whether the detection of each feature point was successful or not based on the feature point detection data stored in the feature point storage unit 33.

[0036] Based on the determination result by the feature point detection success / failure determination processing unit 13, the optimal parameter setting processing unit 14 selects and sets the optimal parameter from among multiple parameter candidates stored in the parameter candidate storage unit 34 for the feature points that were successfully detected. The optimal parameter setting processing unit 14 then saves the set parameter to the parameter storage unit 35.

[0037] The search range setting processing unit 15, based on the determination result by the feature point detection success / failure determination processing unit 13, sets the search range of the template based on the detection position of the successfully detected feature point if feature point detection fails in a given image frame, and if feature point detection was successful in the previous image frame.

[0038] The feature point tracking processing unit 16 uses the template whose parameters have been set by the optimal parameter setting processing unit 14 and the search range setting processing unit 15 to perform a process of tracking feature points in each image frame of the video data acquired thereafter.

[0039] (Operation Example) Next, an operation example of the feature point tracking device CS configured as described above will be explained.

[0040] Figure 4 is a flowchart showing an example of the processing procedure and content of the feature point detection and tracking process performed by the control unit 1 of the feature point tracking device CS.

[0041] (1) When the control unit 1 of the motion image data acquisition feature point tracking device CS receives a detection and tracking start instruction from the management terminal TM and detects this instruction in step S1, in step S2, under the control of the motion image acquisition processing unit 11, it executes the following process for acquiring motion image data from the camera CM.

[0042] Specifically, the video acquisition processing unit 11 activates the camera CM via the sensor I / F unit 4 and receives video data of the monitored user US transmitted from the camera CM via the sensor I / F unit 4. The video acquisition processing unit 11 then stores the received video data in chronological order in the video storage unit 31.

[0043] (2) Feature Point Detection When the acquisition of the video data is started, the control unit 1 of the feature point tracking device CS first performs the following process in step S3 to detect feature points from each image frame of the video data under the control of the feature point detection processing unit 12.

[0044] Figure 5 is a flowchart showing an example of the processing procedure and processing content of the feature point detection process executed by the feature point detection processing unit 12.

[0045] In other words, the feature point detection processing unit 12 first reads image frames one by one from the video storage unit 31 in step S31. Then, in step S32, the feature point detection processing unit 12 inputs the image data of the read image frames as explanatory variables into the feature point detection model stored in the feature point detection model storage unit 32.

[0046] As a result, the feature point detection model processes the image data of the input image frame to detect feature points of the monitored user US, and outputs detection data representing the position of the feature points in X and y coordinates as the target variable. The feature points to be detected are, for example, pixels that make up the image and are distinguished from other pixels by some feature. Features used for distinction include color, shape, and corners.

[0047] In step S33, the feature point detection processing unit 12 acquires feature point detection data from the feature point detection model, and in step S34, stores the acquired feature point detection data in the feature point storage unit 33 while associating it with the frame ID of the source image frame.

[0048] Figure 8 shows an example of human body feature points detected when using the "Openpose" feature point detection model. In this example, 25 feature points are detected, consisting of "nose," "neck," "right shoulder," "right elbow," "right wrist," "left shoulder," "left elbow," "left wrist," "waist," "right waist," "right knee," "right ankle," "waist," "left knee," "left ankle," "right eye," "left eye," "right ear," "left ear," "left big toe," "left little toe," "left heel," "right big toe," "right little toe," and "right heel." Links representing the adjacency relationships between each feature point are also detected. The numbers (0) to (24) in Figure 8 are numbers assigned to identify each of the 25 feature points.

[0049] (3) Setting the optimal template When the feature points are detected in the latest image frame, the control unit 1 of the feature point tracking device CS, under the control of the feature point detection success / failure determination processing unit 13, first reads the detection data of the feature points detected in the latest image frame from the feature point storage unit 33 in step S4. Then, the feature point detection success / failure determination processing unit 13 determines whether or not a feature point has been detected at the position where it should be detected, based on the read feature point detection data.

[0050] As a result of this determination, assume that a feature point was detected at the location where it should have been detected. In this case, the control unit 1 of the feature point tracking device CS proceeds to step S5, and under the control of the optimal parameter setting processing unit 14, executes the following process to set the optimal parameters for the template to be used for tracking feature points thereafter.

[0051] Figure 6 is a flowchart showing an example of the processing procedure and processing content of the optimal parameter setting process executed by the optimal parameter setting processing unit 14.

[0052] Let's assume that the time of the latest image frame is t, and the time of the image frame immediately preceding it is t-1. Let the coordinates of the feature points detected by the feature point detection processing unit 12 in the image frame at time t be (x ｆ , y ｆ Let's assume that this is the case.

[0053] The optimal parameter setting processing unit 14 first sets the detection position of the feature point successfully detected in the image frame at time t-1 as the feature point coordinates in step S51. Next, in step S52, the optimal parameter setting processing unit 14 reads one of the multiple parameter candidates that have been stored in advance from the parameter candidate storage unit 34. Then, in step S53, the optimal parameter setting processing unit 14 attempts to track the feature point for the image frame at time t, using the feature point coordinates as the origin and a template represented by the parameter candidate. Then, when a feature point is detected by the tracking process, the optimal parameter setting processing unit 14 sets the coordinates (x) of the detected feature point. ｔ , y ｔ ) is saved in the working area of ​​the data storage unit 3.

[0054] Once the tracking process using one of the above parameter candidates is completed, the optimal parameter setting processing unit 14 determines in step S54 whether or not all parameter candidates have been selected. If there are still parameter candidates that have not been selected, the optimal parameter setting processing unit 14 returns to step S52 to select the next parameter candidate, and in step S53 attempts to track feature points for the image frame at time t using the selected parameter candidate. Similarly, each time the optimal parameter setting processing unit 14 selects a parameter candidate, it repeatedly attempts to track feature points for the image frame at time t using steps S52 to S54.

[0055] When the tracking process using all parameter candidates is completed, the optimal parameter setting processing unit 14, in step S55, sets the feature point coordinates (x) obtained by the tracking process using each of the parameter candidates. ｔ , y ｔ ) Among these, the feature point coordinates (x) detected by the feature point detection processing unit 12 ｆ , y ｆ The parameter candidate with the smallest distance between the two is selected and stored in the parameter storage unit 35 as the optimal parameter.

[0056] The parameter setting process described above makes it possible to set optimal values ​​for the template parameters. Therefore, by using a template with the above-mentioned optimized parameters, it becomes possible to reliably and stably track feature points even if feature point detection fails in subsequent image frames.

[0057] Incidentally, if parameters are defined, for example, by the width and height representing the template size and the width and height of the search range, if the template size is too small, as shown in Figure 9A, for example, when the position of the hand HD being tracked moves, the matching target of the template TP is more likely to shift to other fingers of the hand HD. Also, if the search range is too small, it becomes impossible to track fast-moving objects.

[0058] On the other hand, if the template size is excessive, the matching target of the template TP may shift from the hand HD (the target of tracking) to the background BB (which has distinctive features such as color or edges), as shown in Figure 9B. Also, if the search range is excessive, it becomes easier for a bug to occur where an object other than the target of tracking is mistakenly selected as the target. For example, even though the left knee of the body was the target of tracking, the target may shift to the right knee.

[0059] In contrast, as shown in Figure 9C, for example, using a template in which the size and search range are optimally set according to the size of the hand HD to be tracked makes it possible to reliably and stably track the target.

[0060] Figure 10 shows examples of errors in feature point detection and feature point tracking when the width and height of the template and the width and height of the search range are changed. In this example, by using a template width and height of 15 pixels and a search range width and height of 20 pixels, which minimize the error, subsequent feature point tracking can be performed stably.

[0061] In the above example, the parameters for the template for feature point tracking were defined using the template width and height, and the search range width and height, but this is not the only way to define them. For example, parameters can also be defined using the radius and intensity of Gaussian blur to reduce noise in the image, or the size of the Laplacian kernel for edge enhancement to improve matching.

[0062] (4) Setting the search range. In step S4 described above, suppose it is determined that feature point detection failed in the image frame at time t. Failure to detect feature points may occur, for example, when a feature point is obscured by the movement of another object.

[0063] In this case, the control unit 1 of the feature point tracking device CS determines in step S6 whether or not the detection of the feature point has succeeded in the image frame at time t-1 one frame before. Then, when it is determined that the detection of the feature point in the image frame at time t-1 has succeeded, the control unit 1 of the feature point tracking device CS, under the control of the search range setting processing unit 15 in step S7, sets the search range of the template used for tracking the feature point in the image frame at time t as follows, based on the coordinates of the feature point detected in the image frame at time t-1.

[0064] For example, let I(t,x,y) be the luminance at coordinates (x, y) in the image frame at time t, and I(t-1,x,y) be the luminance at coordinates (x, y) in the image frame at time t-1 one frame before. Also, let the coordinates of the feature point successfully detected in the image frame at time t-1 be (x ｆ , y ｆ ), let the width and height of the size of the template being set be (t ｗ , t ｈ ), and let the width and height of the search range of the template be (s ｗ , s ｈ ), the search range setting processing unit 15 satisfies x ｆ -s ｗ / 2≦x ｓ ≦x ｆ +s ｗ / 2 y ｆ -s ｈ / 2≦y ｓ ≦y ｆ +s ｈ / 2 In the range of, the error function E is calculated by the following formula, and x that minimizes the error E ｓ , y ｓ is selected.

[0065]

[0066] Here, D is a function representing the color similarity of two pixels, and is expressed as follows. For example, when using the difference of absolute values, D(I １ , I ２ )≒|I １ -I ２It is expressed as |. Also, for example, when using the sum of squared residuals, D(I １ , I ２ ) ≈ √(I １ ２ -I ２ ２ It is expressed as ).

[0067] Note that the functions used to represent the similarity of pixel colors are not limited to the examples above. For example, it is known that OpenCV's match template function allows users to select functions such as TM_SQDIFF, TM_SQDIFF_NORMED, TM_CCORR, TM_CCORR_NORMED, TM_CCOEFF, and TM_CCOEFF_NORMED.

[0068] (5) Tracking Feature Points If the control unit 1 of the feature point tracking device CS fails to detect feature points in an image frame of moving image data, in step S8, under the control of the feature point tracking processing unit 16, it performs feature point tracking using a template whose size and search range are set by the optimal parameters stored in the parameter storage unit 35, or a template whose search range is set by the search range setting processing unit 15.

[0069] Then, when the feature point tracking processing unit 16 finds a feature point through the tracking process, it transmits data representing the tracking result of this feature point from the input / output I / F unit 5 to the management terminal TM. The data representing the tracking result may be stored in a predetermined storage area within the data storage unit 3.

[0070] (6) Determination of the end of detection and tracking The control unit 1 of the feature point tracking device CS monitors in step S9 whether a detection and tracking end instruction has been input from, for example, the management terminal TM. Until a detection and tracking end instruction is input, the series of detection and tracking processes described in steps S2 to S9 are repeatedly executed.

[0071] On the other hand, when a command to end detection and tracking is received, the control unit 1 stops the operation of the camera CM and returns to the standby state.

[0072] (Effects) As described above, in the first embodiment, feature points are detected for each image frame of the video using a feature point detection model employing an existing detection algorithm such as Openpose. If, for example, feature point detection fails in the image frame at time t, and the feature point was successfully detected in the image frame at the previous time t-1, the search range of the template is set based on the coordinates of the feature point detected in the image frame at time t-1, and the feature point of the image frame at time t is tracked using the set template.

[0073] Therefore, even for long videos, the accuracy of feature point detection is maintained, and even if a feature point cannot be detected, the search range for the feature point is set based on the template used for the previous image frame in which a feature point was successfully detected, and the feature point is tracked, making it possible to reliably and accurately detect feature points.

[0074] In the first embodiment, if feature point detection is successful consecutively in the image frame at time t and the image frame immediately preceding it at time t-1, the system attempts to track the feature point in the image frame at time t by sequentially and selectively using a plurality of parameters prepared in advance to define a template, centered on the coordinates of the feature point detected in the image frame at time t-1. The parameter that yields the closest detection position to the feature point successfully detected in the image frame at time t is then stored as the parameter to be used for subsequent feature point tracking.

[0075] Therefore, it is possible to set a template with optimal parameters, and even if feature point detection fails in subsequent image frames, it becomes possible to accurately and effectively track feature points by using the above template.

[0076] [Second Embodiment] In the second embodiment of this invention, information representing the relationships between multiple feature points to be detected is stored, and if the detection of a first feature point fails in a certain image frame, attention is focused on a second feature point adjacent to the first feature point, and if the detection of this second feature point is successful in two consecutive image frames, a search range is set for tracking the first feature point that failed to be detected, based on the movement information of the second feature point between the two image frames.

[0077] In the second embodiment, the configuration of the feature point tracking device CS will also be explained using Figures 2 and 3.

[0078] The feature point tracking device CS stores information representing the relationships between the multiple feature points to be detected in a predetermined storage area within the data storage unit 3. Figure 11 shows the adjacency relationships of the 25 feature points exemplified in Figure 8 in tabular form.

[0079] The feature point tracking device CS performs the search range setting process according to the second embodiment under the control of the search range setting processing unit 15 as follows.

[0080] Figure 7 is a flowchart showing an example of the processing procedure and processing content of the search range setting process executed by the search range setting processing unit 15.

[0081] That is, for example, suppose that the detection of the first feature point failed in the image frame at time t, but was successful in the image frame at time t-1, one frame earlier. On the other hand, suppose that the detection of the second feature point, which is adjacent to the first feature point, was successful in either the image frame at time t or the image frame at time t-1. The adjacent second feature point is identified based on information representing the relationships between multiple feature points stored in the data storage unit 3.

[0082] First, as shown in Figure 7, the search range setting processing unit 15 calculates the amount of movement of the detection position of the second feature point between time t and time t-1 in step S71. Then, in step S72, the search range setting processing unit 15 sets the center position of the search range for tracking the first feature point in the image frame at time t, based on the detection position of the first feature point in the image frame at time t-1 and the calculated amount of movement.

[0083] Let's explain the above process in more detail. For example, suppose we successfully detected the feature point of the "left hip" in the image frame at time t and the image frame at time t-1. In this case, the number of the feature point of the "left hip" is (12) as shown in Figure 8, so the coordinates of the detected position at time t and time t-1 are (x ｆ (t,12), y ｆ (t,12)), (x ｆ (t-1,12), y ｆ This is expressed as (t-1, 12). Also, the amount that the "left hip" moved between time t-1 and time t is (x ｆ (t,12) - x ｆ (t-1,12), y ｆ (t,12)-y ｆ This can be expressed as (t-1,12).

[0084] On the other hand, regarding the feature point "left knee," which is adjacent to the "left hip" mentioned above, suppose detection was successful in the image frame at time t-1, but failed in the image frame at time t. In this case, the coordinates of the detected position of "left knee" at time t-1 are obtained using the feature point number (13), (x ｆ (t-1,13), y ｆ This is expressed as (t-1,13).

[0085] In this state, when using template matching to track the "left knee" at time t, the center of the search range is set to (coordinates of the "left knee" at time t-1) + (amount of movement of the "left hip" from time t-1 to time t).

[0086] In other words, the center of the search range is x ｃ = xｆ (t-1,13) + x ｆ (t,12) - x ｆ (t-1,12) y ｃ = y ｆ (t-1,13) + y ｆ (t,12)-y ｆ Set the time point to (t-1, 12). Then, calculate the error function E using the following formula.

[0087]

[0088] Furthermore, if multiple feature points are defined that are adjacent to the feature point "left knee," the amount of movement is calculated for each of these adjacent feature points, and the center of the search range is set based on the calculated amount of movement for each feature point and the coordinates of the detected position of "left knee" at time t-1.

[0089] For example, suppose that at time t and time t-1, we successfully detected two feature points adjacent to the "left knee": "left hip" (feature point number (12)) and "left ankle" (feature point number (14)). On the other hand, suppose that at time t-1, we successfully detected the feature point adjacent to the "left hip," which is the "left knee," but failed to detect it at time t. In this case, we set the center of the search range as the average of the movement amounts of the "left hip" and the "left ankle."

[0090] In other words, x ｃ = x ｆ (t-1,13) + (x ｆ (t,12) - x ｆ (t-1,12) + x ｆ (t,12) - x ｆ (t-1,14)) / 2 y ｃ = y ｆ (t-1,13) + (y ｆ (t,12)-y ｆ (t-1,12) + y ｆ (t,12)-y ｆ The search range center is set as follows: (t-1,14) / 2.

[0091] As described above, in the second embodiment, information representing the adjacency relationships of multiple feature points to be detected is stored in the data storage unit 3. In this state, if the detection of the first feature point fails in a certain image frame, and if the second feature point adjacent to the first feature point is successfully detected in two consecutive image frames, a search range for tracking the first feature point that failed to be detected is set based on the movement information of the second feature point between the two image frames.

[0092] Therefore, since the search range for tracking feature points that were not detected is set based on the amount of movement of the detection position coordinates of adjacent feature points that were successfully detected in two consecutive image frames, it becomes possible to set the search range for tracking feature points that were not detected simply and accurately.

[0093] [Other Embodiments] (1) When setting the optimal parameters, parameter interpolation may be performed, for example. Possible parameter interpolation methods include interpolation based on the detection result of the previous image frame, linear interpolation or averaging of the detection results of the two most recent frames, and spline interpolation of the detection results of the three most recent frames.

[0094] (2) In addition to Openpose, other existing models may be used as feature point detection models, and the type and structure of the model are not limited to CNN; various existing machine learning models can be used.

[0095] (3) In addition, the configuration of each processing function unit of the feature point tracking device CS, the processing procedures and contents, the types of monitoring targets, the types of feature points, etc., can be modified in various ways without departing from the spirit of this invention.

[0096] Although embodiments of this invention have been described in detail above, the above description is merely illustrative in all respects. It goes without saying that various improvements and modifications can be made without departing from the scope of this invention. In other words, when implementing this invention, specific configurations may be adopted as appropriate depending on the embodiment.

[0097] In short, this invention is not limited to the embodiments described above, and the components can be modified and implemented in practice without departing from the gist of the invention. Furthermore, various inventions can be formed by appropriately combining the multiple components disclosed in each embodiment. For example, some components may be deleted from all the components shown in each embodiment. Moreover, components from different embodiments may be appropriately combined.

[0098] Furthermore, the functions of the elements disclosed herein may be implemented using circuits or processing circuitry that include general-purpose processors, special-purpose processors, integrated circuits, ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), conventional circuits, and / or combinations thereof that are programmed using one or more programs stored in one or more memories, or otherwise configured to perform the disclosed functions. A processor is considered processing circuitry or circuitry because it includes transistors and other circuits. A processor may be a programmed processor that executes programs stored in memory. In this disclosure, a circuit, unit, or means is hardware that performs the enumerated functions, or hardware programmed to perform the enumerated functions. Hardware may be any hardware disclosed herein that is programmed or configured to perform the enumerated functions.

[0099] Furthermore, there is memory for storing computer programs that include computer instructions. These computer instructions provide logic and routines that enable hardware (e.g., processing circuitry or circuitry) to perform the methods disclosed herein. This computer program can be implemented in commonly known forms, such as computer-readable storage media, computer program products, memory devices, recording media such as CD-ROMs and DVDs, and / or memory in FPGAs and ASICs.

[0100] CS: Feature point tracking device CM: Camera TM: Management terminal NW: Network US: Monitored user 1: Control unit 2: Program storage unit 3: Data storage unit 4: Sensor I / F unit 5: Input / output I / F unit 6: Bus 11: Video acquisition processing unit 12: Feature point detection processing unit 13: Feature point detection success / failure determination processing unit 14: Optimal parameter setting processing unit 15: Search range setting processing unit 31: Video storage unit 32: Feature point detection model storage unit 33: Feature point storage unit 34: Parameter candidate storage unit 35: Parameter storage unit

Claims

1. A feature point tracking device comprising: a first processing unit that determines whether or not feature point detection has been successful in a first image frame among a plurality of consecutive image frames in a time series contained in a video; a second processing unit that determines whether or not feature point detection has been successful in a second image frame located earlier in the time series than the first image frame if it is determined that feature point detection has failed in the first image frame; a third processing unit that sets template parameters based on the detection position of the feature point detected in the second image frame if it is determined that feature point detection has been successful in the second image frame; and a fourth processing unit that tracks the feature point in the first image frame using the template represented by the reset parameters.

2. The feature point tracking device according to claim 1, further comprising: a fifth processing unit which, when it is determined that the detection of the feature point in the first image frame has been successful, selectively uses a plurality of templates with different parameters to track the feature point in the first image frame; and a sixth processing unit which identifies the parameters that minimize the distance between the tracking position of the feature point obtained by the tracking and the detection position of the feature point detected from the first image frame, and sets the template represented by the identified parameters as a template for tracking the feature point.

3. The feature point tracking device according to claim 1, wherein the first processing unit determines whether it has succeeded in detecting a first feature point and a second feature point having a predetermined relationship with the first feature point in the first image frame; if the second processing unit determines that it has succeeded in detecting the second feature point and failed in detecting the first feature point, it determines whether it has succeeded in detecting the first feature point and the second feature point in the second image frame which is located earlier in the time series than the first image frame; and if the second processing unit determines that it has succeeded in detecting the first feature point and the second feature point, the third processing unit sets the parameters of the template for tracking the first feature point in the first image frame based on the movement information of the detection position of the second feature point between the second image frame and the first image frame and the detection position of the first feature point in the second image frame.

4. A program that causes a processor in a feature point tracking device to perform the processing performed by each processing unit in the feature point tracking device according to any one of claims 1 to 3.

Images