Method and system for tracking multiple objects based on deep learning by considering directional information of objects

Abstract

The present invention relates to a method and system for tracking multiple objects based on deep learning by considering directional information of objects, which: generate a region of interest in an image captured on a road; derive an intersection angle on the basis of the region of interest; and when there is a re-identified vehicle object while tracking a plurality of vehicle objects on the road, correct location coordinates of the corresponding vehicle object according to the region of interest and the intersection angle and continue tracking.

Description

Deep learning-based multi-object tracking method and system considering object orientation information

[0001] The present invention relates to a deep learning-based multi-object tracking method and system that considers the directional information of an object, wherein a region of interest is generated from an image captured on a road, an intersection angle is derived based on the region of interest, and when a vehicle object is re-identified while tracking a plurality of vehicle objects on the road, the location coordinates of the vehicle object are corrected according to the region of interest and the intersection angle, and tracking is continued.

[0002]

[0003] With the recent introduction and advancement of deep learning technology, the accuracy of object detection has significantly improved, leading to the development of object tracking technology based on it. In particular, multi-object tracking has emerged as a technology for simultaneously identifying and tracking multiple objects in environments where multiple objects intersect or move in complex ways.

[0004] However, in multi-object tracking, to improve tracking accuracy when objects disappear from camera footage due to obscuration by obstacles and reappear, or when multiple cameras capture different areas, it is necessary to re-identify whether the previously tracked object and the currently tracked object are the same. In particular, the SORT (Simple Online Realtime Tracking) algorithm, developed for real-time object tracking, can track the trajectory of an object in real time by utilizing the output of an object detector (e.g., a bounding box). However, the performance of the SORT algorithm may be limited in complex environments, such as when overlapping occurs between multiple objects.

[0005] Conventional multi-object tracking technologies include an efficient multi-object tracking method and apparatus based on online restricted learning of entire object models and model states, as disclosed in Korean Published Patent No. 10-2022-0156062. The invention detects the appearance features of objects detected in consecutive frames, predicts the future trajectories of the detected objects, and distinguishes and links the detected objects based on the appearance features and future trajectories. However, in conventional multi-object tracking technologies, when multiple various objects moving in real time exist within a frame, the accuracy of object tracking may be degraded due to overlapping or occlusion.

[0006] Therefore, there is a need to develop technology that can improve the accuracy of multi-object tracking.

[0007]

[0008] The present invention aims to provide a deep learning-based multi-object tracking method and system that considers the directional information of objects, wherein the method generates a region of interest in an image captured on a road, derives an intersection angle based on the region of interest, and, when a vehicle object is re-identified while tracking multiple vehicle objects on the road, corrects the position coordinates of the vehicle object according to the region of interest and the intersection angle and continues tracking.

[0009]

[0010] In order to solve the above-mentioned problem, one embodiment of the present invention provides a deep learning-based multi-object tracking method considering directional information of an object, performed in a computing system comprising one or more processors and one or more memories, comprising: a region of interest generation step of generating a region of interest having at least four vertices on a road based on a video captured on a road according to region of interest (ROI) setting information directly set by a person; an intersection angle derivation step of deriving an intersection angle formed by the intersection of diagonals connecting a plurality of vertices included in the region of interest; an object tracking step of deriving tracking information of an object captured in the video; and a correction position derivation step of deriving a corrected position of an object re-identified in the object tracking step by correcting the re-identified position of the object based on the ROI setting information. The method provides a multi-object tracking method comprising: a final position derivation step for deriving the final position of an object re-identified in the object tracking step by correcting the re-identification position or correction position of the object re-identified in the object tracking step based on the intersection angle, wherein the intersection angle represents the maximum directionality that determines the path the object can move on the road.

[0011] In one embodiment of the present invention, the object tracking step may include: an object derivation step of deriving first position information of each of a plurality of first objects included in a past frame prior to the current frame of the image and second position information of each of a plurality of second objects included in the current frame through a deep learning-based detection model; a predicted position derivation step of deriving a predicted position at a time corresponding to the current frame for each of the plurality of first objects through a pre-set prediction algorithm based on the first position information; an IOU calculation step of calculating an IOU (Intersection Over Union) value for a bounding box for the predicted position of each of the plurality of first objects and a bounding box for the current position included in the second position information of each of the plurality of second objects; and a re-identification position derivation step of deriving the tracking information including the re-identification position of a second object re-identified as the same object as the first object by applying a plurality of IOU values ​​to a Hungarian algorithm.

[0012] In one embodiment of the present invention, the predicted position derivation step may include: an information input step of applying coordinate information, velocity information, and a time difference value between the current frame and the past frame included in the first position information to the prediction algorithm for each of the plurality of first objects; and a first object position derivation step of deriving the position coordinates at a time corresponding to the current frame predicted by the prediction algorithm as the predicted position for each of the plurality of first objects.

[0013] In one embodiment of the present invention, the re-identification location derivation step may include: a re-identification object extraction step for extracting a second object that is re-identified by applying the plurality of IOU values ​​to a Hungarian algorithm and determining that it is identical to a first object; an object re-identification step for mapping the ID (Identifier) ​​of the re-identified second object to the corresponding ID of the first object; and a re-identification location determination step for determining the current location derived for the re-identified second object as the re-identification location.

[0014] In one embodiment of the present invention, the ROI setting information may include the coordinate positions of each of at least four vertices forming the boundary of the region of interest, and an analysis target within the region of interest.

[0015] In one embodiment of the present invention, the intersection angle derivation step may include: a diagonal generation step for generating a diagonal connecting opposite vertices among a plurality of vertices included in the region of interest; and an intersection angle selection step for selecting and deriving an intersection angle corresponding to the direction of travel of the road among a plurality of intersection angles formed by the intersection of the generated plurality of diagonals.

[0016] In one embodiment of the present invention, the correction position derivation step derives the correction position by deriving the position coordinates within the boundary of the region of interest that are closest to the re-identification position when the position coordinates of the re-identification position are outside the range of the region of interest, and the correction position may be determined as the position coordinates where the shortest distance between the re-identification position and the boundary of the region of interest is formed.

[0017] In one embodiment of the present invention, the final position derivation step comprises: a positional area generation step for the re-identified object, which generates a positional area of ​​the re-identified object within the region of interest based on the position information of the object tracked in a past frame and the intersection angle; and a final position determination step for determining the position coordinate within the boundary of the positional area closest to the re-identified position or the correction position as the final position when the position coordinate of the re-identified position or the correction position is outside the range of the positional area; wherein the final position may be determined as the position coordinate where the shortest distance between the re-identified position or the correction position and the boundary of the positional area is formed.

[0018] In order to solve the above-mentioned problem, one embodiment of the present invention comprises a computing system that performs a deep learning-based multi-object tracking method considering the directional information of an object, including one or more processors and one or more memories, and comprises: an interest region generation unit that generates an interest region having at least four vertices on the road according to ROI (Region Of Interest) setting information directly set by a person based on an image captured on the road; an intersection angle derivation unit that derives an intersection angle formed by the intersection of diagonals connecting a plurality of vertices included in the interest region; an object tracking unit that derives tracking information of an object captured in the image; and a correction position derivation unit that derives a correction position of the re-identified object by correcting the re-identified position of the object tracking unit based on the ROI setting information. The present invention provides a computing system comprising: a final position derivation unit that derives the final position of an object re-identified by correcting the re-identification position or correction position of the object re-identified by the object tracking unit based on the intersection angle, wherein the intersection angle represents the maximum directionality that determines the path the object can move on the road.

[0019]

[0020] In one embodiment of the present invention, by using mathematical algorithms to re-identify objects whose tracking has been interrupted for a plurality of objects detected in each of the past frame and the current frame, the effect of improving the tracking accuracy of multiple objects captured in the image can be achieved.

[0021] In one embodiment of the present invention, by deriving the maximum direction in which an object located within a road can move based on a region of interest set on the road, the tracking accuracy of the movement of a re-identified object can be improved.

[0022] In one embodiment of the present invention, by correcting the position coordinates of an object moving within a road based on a region of interest and an intersection angle generated with respect to the road, the tracking accuracy of the movement of a re-identified object can be improved.

[0023] In one embodiment of the present invention, by generating a positional area where the object can move based on the previous time-based location of the re-identified object, the effect of improving the tracking accuracy of the movement of the re-identified object can be achieved.

[0024] In one embodiment of the present invention, when two or more roads with different directions of travel are captured in an image, the position coordinates at the current time point are corrected by considering the region of interest generated from the road corresponding to the position coordinates at the past time point of the re-identified object and the intersection angle, thereby improving the tracking accuracy of the movement of the re-identified object.

[0025]

[0026] FIG. 1 schematically illustrates the internal configuration of a computing system performing a multi-object tracking method according to an embodiment of the present invention and the steps for performing a multi-object tracking method.

[0027] FIG. 2 schematically illustrates a tracking flow for detected multiple objects according to one embodiment of the present invention.

[0028] FIG. 3 schematically illustrates the process of deriving the predicted position of a first object detected in a past frame according to an embodiment of the present invention.

[0029] FIG. 4 schematically illustrates the process of deriving the current position of a second object detected in a current frame according to an embodiment of the present invention.

[0030] FIG. 5 schematically illustrates the process of performing the re-identification location derivation step according to one embodiment of the present invention.

[0031] FIG. 6 schematically illustrates ROI setting information and region of interest according to one embodiment of the present invention.

[0032] FIG. 7 schematically illustrates the intersection angle derived within the region of interest according to one embodiment of the present invention.

[0033] FIG. 8 schematically illustrates the process of determining the direction in which the intersection angle faces according to one embodiment of the present invention.

[0034] FIG. 9 schematically illustrates the process of performing the correction position derivation step according to one embodiment of the present invention.

[0035] FIG. 10 schematically illustrates the process of performing the final position derivation step according to one embodiment of the present invention.

[0036] FIG. 11 illustrates, in an exemplary manner, the internal configuration of a computing device according to one embodiment of the present invention.

[0037] Hereinafter, various embodiments and / or aspects are disclosed with reference to the drawings. For illustrative purposes, numerous specific details are disclosed in the following description to aid in a general understanding of one or more aspects. However, it will also be recognized by those skilled in the art that these aspects may be practiced without such specific details. The following description and the accompanying drawings describe specific exemplary aspects of one or more aspects in detail. However, these aspects are exemplary, and some of the various methods in the principles of the various aspects may be used, and the description is intended to include all such aspects and their equivalents.

[0038]

[0039] In addition, various aspects and features will be presented by a system that may include multiple devices, components and / or modules, etc. It should also be understood and recognized that various systems may include additional devices, components and / or modules, etc., and / or may not include all of the devices, components, modules, etc. discussed in relation to the drawings.

[0040] Terms such as “embodiment,” “example,” “aspect,” “example,” etc. as used herein may not be interpreted as implying that any aspect or design described is superior or more advantageous than other aspects or designs. Terms used below, such as “part,” “component,” “module,” “system,” “interface,” etc., generally refer to computer-related entities and may, for example, refer to hardware, a combination of hardware and software, or software.

[0041] Additionally, the terms “comprising” and / or “comprising” should be understood to mean that the relevant feature and / or component is present, but not to exclude the presence or addition of one or more other features, components and / or groups thereof.

[0042] Additionally, terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. Such terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0043] Furthermore, in the embodiments of the present invention, all terms used herein, including technical or scientific terms, unless otherwise defined, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in the embodiments of the present invention.

[0044]

[0045] FIG. 1 schematically illustrates the internal configuration of a computing system (1000) that performs a multi-object tracking method according to an embodiment of the present invention and the steps for performing a multi-object tracking method.

[0046]

[0047] As illustrated in FIG. 1, a computing system (1000) comprising one or more processors and one or more memories, and performing a deep learning-based multi-object tracking method considering the directional information of an object, comprises: an interest region generation unit (1100) that generates an interest region having at least four vertices on the road according to ROI (Region Of Interest) setting information directly set by a person based on an image captured on the road; an intersection angle derivation unit (1200) that derives an intersection angle formed by the intersection of diagonals connecting a plurality of vertices included in the interest region; an object tracking unit (1300) that derives tracking information of an object captured in the image; and a correction position derivation unit (1400) that derives a correction position of the re-identified object by correcting the re-identified position of the object tracking unit (1300) based on the ROI setting information. and a final position derivation unit (1500) that derives the final position of the re-identified object by correcting the re-identification position or correction position of the re-identified object in the object tracking unit (1300) based on the above intersection angle; wherein the intersection angle represents the maximum directionality that determines the path the object can move on the road.

[0048] Additionally, a deep learning-based multi-object tracking method considering directional information of an object, performed in a computing system (1000) comprising one or more processors and one or more memories, comprising: a region of interest generation step (S100) for generating a region of interest having at least four vertices on the road according to region of interest (ROI) setting information directly set by a person based on an image captured on the road; an intersection angle derivation step (S200) for deriving an intersection angle formed by the intersection of diagonals connecting a plurality of vertices included in the region of interest; an object tracking step (S300) for deriving tracking information of an object captured in the image; and a correction position derivation step (S400) for deriving a correction position of an object re-identified in the object tracking step (S300) by correcting the re-identified position of the object based on the ROI setting information. and a final position derivation step (S500) for deriving the final position of the re-identified object by correcting the re-identification position or correction position of the re-identified object in the object tracking step (S300) based on the intersection angle, wherein the intersection angle represents the maximum directionality that determines the path the object can move on the road.

[0049]

[0050] Schematically, FIG. 1 (a) illustrates the internal configuration of a computing system (1000) that performs a multi-object tracking method, and FIG. 1 (b) illustrates the steps for performing a multi-object tracking method.

[0051]

[0052] Specifically, a computing system (1000) performing the multi-object tracking method of the present invention is connected wirelessly or via a wired connection to an external camera that photographs a road and acquires an image of the road captured by said camera, and said computing system (1000) can track a plurality of vehicle objects captured in said image. As an embodiment of the present invention, the objects described below may be limited to vehicle objects located on the road.

[0053] Generally, situations may arise where an excessive number of objects within an image frame obscure each other's appearances, or where object tracking is interrupted by obstacles. Alternatively, when multiple cameras capturing a moving object divide a single area, an object that moves out of its respective shooting area may be recognized as a different object by each camera, potentially causing object tracking to be lost.

[0054] The present invention re-identifies an object to prevent such interruption in object tracking, and corrects the location of the re-identified object based on the location coordinates of the object captured in a past frame to improve the tracking accuracy of the re-identified object.

[0055] More specifically, as illustrated in FIG. 1(a), the computing system (1000) includes an area of ​​interest generation unit (1100), an intersection angle derivation unit (1200), an object tracking unit (1300), a correction position derivation unit (1400), and a final position derivation unit (1500). The area of ​​interest generation unit (1100) generates an area of ​​interest on a road based on ROI setting information directly set by a person (e.g., an administrator of the computing system) that is input to the computing system (1000). The final position derivation unit (1500) tracks multiple objects re-identified in the image in real time. Depending on the embodiment, the tracking results may be transmitted to an external server or managed directly, and may be visually output to a display connected to the computing system (1000) so that a controller can monitor them.

[0056]

[0057] As illustrated in FIG. 1(b), the region of interest generation step (S100) is performed by the region of interest generation unit (1100), and a region of interest having at least four vertices on the road is generated according to ROI setting information directly set by a person based on an image taken on the road. In one embodiment of the present invention, the ROI setting information may include the location coordinates of each of the plurality of vertices included in the region of interest.

[0058] The intersection angle derivation step (S200) is performed by the intersection angle derivation unit (1200) and derives an intersection angle formed by the intersection of diagonals connecting multiple vertices included in the area of ​​interest. When the area of ​​interest generated on the road has four vertices, an intersection angle may be formed by two diagonals, and the intersection angle represents the maximum directionality that determines the path an object can move on the road.

[0059] The object tracking step (S300) is performed by the object tracking unit (1300) and derives tracking information for a plurality of objects captured in each of the current frame and the past frame prior to the current frame captured in the image. More specifically, the object tracking unit (1300) determines whether the object of the past frame and the object of the current frame match based on the location information of each object detected in the past frame and the current frame through the object tracking step (S300), and can match the ID (Identifier) ​​of the object of the past frame and the object of the current frame by re-identifying the object of the current frame. Preferably, as an embodiment of the present invention, object re-identification can be implemented by combining a SORT (Simple Online and Realtime Tracking) algorithm with a deep learning-based object re-identification model. Meanwhile, in an embodiment of the present invention, the tracking information may include frame-by-frame location coordinates and identified IDs for objects captured in the image.

[0060] The correction position derivation step (S400) is performed by the correction position derivation unit (1400), and based on the ROI setting information, the re-identification position of the object re-identified in the object tracking step (S300) is corrected to derive the correct position of the re-identified object. It is preferable that the vehicle object corresponding to the tracking target moves on the road, and the present invention can improve the accuracy of multi-object tracking by correcting the position coordinates of the re-identified object based on the region of interest created on the road.

[0061] The final position derivation step (S500) is performed by the final position derivation unit (1500), and based on the intersection angle, the final position of the re-identified object is derived by correcting the re-identified position of the object re-identified in the object tracking step (S300) or the corrected position derived in the correction position derivation step (S400). More specifically, the present invention can improve the accuracy of multi-object tracking by generating an area where the object can be located at the current time based on the intersection angle and based on the past time point location of the re-identified object, and correcting the position coordinates of the re-identified object based on that area.

[0062]

[0063] In other words, the present invention is characterized by the technical feature of being able to re-identify an object that has lost tracking in a past frame and a current frame, and improving the accuracy of multi-object tracking by correcting the location of the re-identified object based on the location in the past frame and the environment (region of interest, and intersection angle) where the object is located.

[0064]

[0065] FIG. 2 schematically illustrates a tracking flow for detected multiple objects according to one embodiment of the present invention.

[0066]

[0067] As illustrated in FIG. 2, the object tracking step (S300) comprises: an object derivation step that derives first position information of each of a plurality of first objects included in a past frame prior to the current frame of the image and second position information of each of a plurality of second objects included in the current frame through a deep learning-based detection model; a predicted position derivation step that derives a predicted position at a time corresponding to the current frame for each of the plurality of first objects through a pre-set prediction algorithm based on the first position information; an IOU calculation step that calculates an IOU (Intersection Over Union) value for a bounding box for the predicted position of each of the plurality of first objects and a bounding box for the current position included in the second position information of each of the plurality of second objects; and a re-identification position derivation step that derives the tracking information including the re-identification position of a second object re-identified as the same object as the first object by applying the plurality of IOU values ​​to a Hungarian algorithm.

[0068]

[0069] Specifically, the object tracking step (S300) includes an object extraction step, a predicted location extraction step, an IOU calculation step, and a re-identification location extraction step. When an object captured in a past frame of an image is referred to as a first object and an object captured in a current frame is referred to as a second object, the object extraction step derives first location information for each of a plurality of first objects included in the past frame and second location information for each of a plurality of second objects included in the current frame through a deep learning-based detection model. As an embodiment of the present invention, the detection model may include an artificial neural network such as a Convolutional Neural Network (CNN) or a Deep Neural Network (DNN), and may detect an object in an image, generate a bounding box surrounding the detected object, and derive location information including location coordinates, size information, and velocity information for the bounding box.

[0070] The above-described predicted position derivation step derives, for each of the plurality of first objects, a predicted position at a time corresponding to the current frame based on the first position information through a preset prediction algorithm. As in an embodiment illustrated in FIG. 2, the preset prediction algorithm may include a Kalman filter.

[0071] The above IOU calculation step calculates an IOU (Intersection Over Union) value for a bounding box for the predicted position of each of the plurality of first objects and a bounding box for the current position included in the second position information of each of the plurality of second objects. Preferably, the IOU value is an indicator that shows how much two object regions overlap within an image, and the higher the calculated IOU value, the higher the similarity between the first object and the second object associated with the IOU value.

[0072] The above re-identification location derivation step derives the tracking information including the re-identification location of a second object that is re-identified as the same object as the first object by applying a plurality of IOU values ​​to the Hungarian Algorithm. More specifically, by applying the IOU values ​​calculated for a pair of the first object and the second object to the Hungarian Algorithm, consistency in multi-object tracking is maintained, thereby enabling highly reliable object re-identification.

[0073] Subsequently, a detailed explanation will be provided regarding the method of correcting the location coordinates of a re-identified object based on the region of interest and the intersection angle, and continuing to track the object.

[0074]

[0075] FIG. 3 schematically illustrates the process of deriving the predicted position of a first object detected in a past frame according to an embodiment of the present invention. FIG. 4 also schematically illustrates the process of deriving the current position of a second object detected in a current frame according to an embodiment of the present invention.

[0076]

[0077] As illustrated in FIG. 3, the predicted position derivation step comprises: an information input step for each of the plurality of first objects, applying coordinate information, velocity information, and the time difference value between the current frame and the past frame included in the first position information to the prediction algorithm; and a first object position derivation step for each of the plurality of first objects, deriving the position coordinates at the time corresponding to the current frame predicted by the prediction algorithm as the predicted position.

[0078] Additionally, as illustrated in FIG. 4, the object tracking step (S300) includes an object derivation step that derives first position information of each of a plurality of first objects included in a past frame prior to the current frame of the image and second position information of each of a plurality of second objects included in the current frame through a deep learning-based detection model.

[0079]

[0080] Specifically, as illustrated in FIG. 3, the predicted position derivation step includes an information input step and a first object position derivation step. The information input step inputs coordinate information, velocity information, and a time difference value between the current frame and the past frame, which are included in the first position information for each of the plurality of first objects derived from the past frame through the deep learning-based detection model, to the prediction algorithm (e.g., Kalman filter). In an embodiment of the present invention illustrated in FIG. 3, the coordinate information may include the 3D position coordinates (x1, y1, h1) of a bounding box generated for the first object at a time point corresponding to the past frame, and the size information of the bounding box (e.g., aspect ratio r1). The velocity information may include the degree of change over time for each element included in the coordinate information (vx1, vy1, vr1, vh1). Additionally, the time difference value between the current frame and the past frame may be calculated as T2-T1.

[0081] Preferably, velocity information can be generated based on changes in bounding boxes for objects that maintained a track in multiple past frames taken at a point in time prior to the current frame. More preferably, the current frame can be determined as an image frame at the point in time when the tracking of the object is interrupted, and since a first position information for each object can be generated by maintaining a single track for each object in multiple past frames prior to the current frame, each of the multiple first objects can be treated identically to each of the multiple tracks formed in the past frames.

[0082] The first object position derivation step derives the position coordinates at a time corresponding to the current frame, predicted by the prediction algorithm, as the predicted position for each of the plurality of first objects. As an embodiment of the present invention, the predicted position may include position coordinates (x1', y1', h1') and size information (r1') predicted for the first object at a time corresponding to the current frame.

[0083]

[0084] In addition, as shown in FIG. 4, a plurality of second objects are detected in the current frame through the deep learning-based detection model, and second position information including the current position (x2, y2, h2) and size information (r2) for each of the plurality of second objects can be derived.

[0085] The present invention compares a predicted object and an actual object in the same time period and can re-identify an object in the current frame based on the comparison result.

[0086]

[0087] FIG. 5 schematically illustrates the process of performing the re-identification location derivation step according to one embodiment of the present invention.

[0088]

[0089] As illustrated in FIG. 5, the re-identification location derivation step comprises: a re-identification object extraction step for extracting a second object that is re-identified by applying the plurality of IOU values ​​to a Hungarian algorithm and determining that it is identical to the first object; an object re-identification step for mapping the ID (Identifier) ​​of the re-identified second object to the corresponding ID of the first object; and a re-identification location determination step for determining the current location derived for the re-identified second object as the re-identification location.

[0090]

[0091] Schematically, FIG. 5(a) illustrates a plurality of IOU values ​​generated for a plurality of first objects included in a past frame and a plurality of second objects included in a current frame, and FIG. 5(b) illustrates the process of performing an object re-identification step.

[0092]

[0093] Specifically, the re-identification location derivation step includes a re-identification object extraction step, an object re-identification step, and a re-identification location determination step, wherein the re-identification object extraction step applies a plurality of IOU values ​​to the Hungarian algorithm to extract a second object that is re-identified as identical to the first object. As described above in FIG. 3, since one track is formed for each first object included in the past frame, when four first objects are identified in the past frame and four second objects are identified in the current frame, an IOU value can be calculated for each pair of first objects (Track) and second objects (Object), as shown in the table in FIG. 5 (a).

[0094] The Hungarian algorithm is an algorithm for solving the minimum cost matching problem. In the present invention, the IOU value representing the similarity of bounding boxes between a first object and a second object is converted into a cost matrix using the Hungarian algorithm, and based on the matrix, the second object corresponding to the first object can be found.

[0095] Preferably, in one embodiment of the present invention, by applying (1-IOU value) to the Hungarian algorithm for each pair between a first object and a second object, the first object and the second object for which the minimum value is finally calculated are determined to be the same object, and the second object that is re-identified corresponding to each of the plurality of first objects is extracted. As in one embodiment illustrated in FIG. 5(a), depending on the result of applying the plurality of IOU values ​​to the Hungarian algorithm, the first object #1 and the second object #2 may be re-identified as the same object, the first object #2 and the second object #3 may be re-identified as the same object, the first object #3 and the second object #4 may be re-identified as the same object, and the first object #4 and the second object #1 may be re-identified as the same object.

[0096] As illustrated in FIG. 5(b), the object re-identification step performs Re-ID by mapping the ID (Identifier) ​​of the re-identified second object extracted through the re-identification object extraction step to the ID of the first object determined to be the same object as the second object.

[0097] Meanwhile, the re-identification location determination step determines the current location derived for the second object being re-identified as the re-identification location, and the computing system (1000) subsequently corrects the re-identification location based on the region of interest and intersection angle generated on the road where the second object was identified.

[0098]

[0099] FIG. 6 schematically illustrates ROI setting information and region of interest according to one embodiment of the present invention.

[0100]

[0101] As illustrated in FIG. 6, the ROI setting information includes the coordinate positions of each of at least four vertices forming the boundary of the region of interest, and an analysis target within the region of interest. Additionally, as a deep learning-based multi-object tracking method, it includes a region of interest generation step (S100) of generating a region of interest having at least four vertices on the road according to the ROI (Region Of Interest) setting information directly set by a person based on an image captured on the road.

[0102]

[0103] Specifically, as described above in FIG. 2, a user using the computing system (1000) of the present invention can generate a region of interest on a road captured in an image based on the image. Preferably, the computing system (1000) can generate a region of interest on a road based on data included in ROI setting information directly set by the user based on the image through the region of interest generation step (S100).

[0104]

[0105] In one embodiment of the present invention, the ROI setting information may include the coordinate positions of each of at least four vertices forming the boundary of the region of interest, and an analysis target (e.g., a vehicle object) within the region of interest, and in another embodiment, may further include settings for the region of interest such as the shape of the region of interest (e.g., the boundary of the region of interest formed in a curved shape).

[0106] More specifically, since the region of interest of the present invention corresponds to an area for tracking vehicle objects moving on a road, it can be generated to correspond to the shape of the road captured in the image, and in one embodiment of the present invention, it is more preferable to generate it in a form having at least four vertices, such as region of interest #1 and region of interest #2 shown in FIG. 6, in order to form an intersection angle corresponding to the movable direction of the vehicle object.

[0107]

[0108] FIG. 7 schematically illustrates the intersection angle derived within the region of interest according to one embodiment of the present invention.

[0109]

[0110] As illustrated in FIG. 7, the intersection angle derivation step (S200) includes: a diagonal generation step for generating a diagonal connecting opposite vertices among a plurality of vertices included in the region of interest; and an intersection angle selection step for selecting and deriving an intersection angle corresponding to the direction of travel of the road among a plurality of intersection angles formed by the intersection of the generated plurality of diagonals.

[0111]

[0112] Specifically, the intersection angle derivation step (S200) includes a diagonal generation step and an intersection angle selection step, and the computing system (1000) can generate a diagonal by connecting opposite vertices among at least four vertices included in the region of interest through the diagonal generation step. Preferably, as in an embodiment shown in FIG. 7, two diagonals can be generated for each of the region of interest #1 and region of interest #2 having four vertices.

[0113] Preferably, when two diagonals intersect, a total of four candidate intersection angles can be formed for each different direction, and the computing system (1000) selects and derives an intersection angle corresponding to the direction of travel of the road among the multiple intersection angles formed by the intersection of multiple diagonals generated through the intersection angle selection step. As in an embodiment shown in FIG. 7, an intersection angle θ1 corresponding to the direction of travel of the road included in region of interest #1 is selected and derived, and an intersection angle θ2 corresponding to the direction of travel of the road included in region of interest #2 can be selected and derived.

[0114] Additionally, the final position derivation step (S500) includes a positional area generation step for the re-identified object, which generates a positional area of ​​the re-identified object within the interest area based on the position information of the object tracked in the past frame and the intersection angle. In an embodiment illustrated in FIG. 7, if the intersection point of two diagonals generated within interest area #1 corresponds to the position coordinates of the re-identified object at a past point in time, a positional area corresponding to S1 is generated, and if the intersection point of two diagonals generated within interest area #2 corresponds to the position coordinates of the re-identified object at a past point in time, a positional area corresponding to S2 can be generated.

[0115]

[0116] Meanwhile, as previously described, the above intersection angle represents the maximum directionality that determines the path an object can move on the road, and the present invention determines that the movement of an object detected in a region of interest cannot deviate from the intersection angle derived from the region of interest, and accordingly, by correcting the detected position coordinates of the object, it can produce the effect of improving the detection accuracy of the object.

[0117]

[0118] FIG. 8 schematically illustrates the process of determining the direction in which the intersection angle faces according to one embodiment of the present invention.

[0119]

[0120] Schematically, FIG. 8(a) illustrates the direction of the intersection angle determined in one embodiment, and FIG. 8(b) illustrates the direction of the intersection angle determined in another embodiment.

[0121]

[0122] Specifically, in the case of a vehicle object moving on a road, it is desirable that the area where it can currently be located be determined according to the movement path at a previous point in time. Therefore, the present invention can determine that the direction of the intersection angle generated based on the position coordinates detected at a point in time corresponding to a past frame of the re-identified object corresponds to the direction of movement of the object at the point in time corresponding to the past frame.

[0123] That is, as in the embodiments illustrated in FIG. 8 (a) and (b), when the position coordinates of an object detected in a past frame are P1(x1, y1) and the position coordinates of an object detected in a frame earlier than the past frame are P0(x0, y0), the position area of ​​the object that can be located at the current time point can be determined according to the movement vector (x0-x1, y0-y1) of the object at the past time point.

[0124]

[0125] FIG. 9 schematically illustrates the process of performing the correction position derivation step (S400) according to one embodiment of the present invention.

[0126]

[0127] As illustrated in FIG. 9, the correction position derivation step (S400) derives the correction position by finding the position coordinates within the boundary of the area of ​​interest that are closest to the re-identified position when the position coordinates of the re-identified position are outside the range of the area of ​​interest, and the correction position is determined as the position coordinates where the shortest distance between the re-identified position and the boundary of the area of ​​interest is formed.

[0128]

[0129] Schematically, FIG. 9(a) illustrates the process of deriving the correction position, and FIG. 9(b) illustrates the region of interest applied according to the position of the object.

[0130]

[0131] Specifically, in the present invention, it is preferable that the re-identified object is moving within the region of interest. Accordingly, if the location coordinates of the re-identified object are detected outside the region of interest, the present invention improves the detection accuracy of the object by correcting the location coordinates of the re-identified object so that they are located within the region of interest.

[0132]

[0133] As illustrated in FIG. 9(a), the re-identified location of the re-identified object (P1) derived by the re-identified location derivation step is (x1, y1), and if the re-identified location is derived outside the region of interest, the computing system (1000) derives the location coordinate P1'(x1', y1') as the corrected location, which is the location coordinate within the boundary line closest to the re-identified location in the region of interest, where the shortest distance between the re-identified location and the boundary line is formed, through the correction location derivation step (S400).

[0134] In other words, for a re-identified object, the position coordinates corresponding to the foot of the perpendicular drawn from the region of interest to the boundary line closest to the re-identified object are derived as the correction position.

[0135]

[0136] Meanwhile, in the correction position derivation step (S400), if the re-identification position of the re-identified object derived by the re-identification position derivation step is derived within the region of interest, it is preferable to derive the position coordinates of the re-identification position as the correction position as is.

[0137]

[0138] Additionally, as shown in the embodiment illustrated in FIG. 9(b), the computing system (1000) can perform multi-object tracking based on images of two roads with different directions of travel. In the case of the embodiment, it is preferable that different regions of interest are generated on each of the two roads with different directions of travel, and since the position of an object is corrected based on ROI setting information for the region of interest, the computing system (1000) corrects the current position of the object based on the region of interest including the position coordinates in the past frame for the re-identified object.

[0139] That is, as illustrated in Fig. 9(b), it is desirable that the re-identification location of a re-identified object located at P1(x1, y1) at a time corresponding to a past frame be corrected based on the ROI setting information for region of interest #1, and the re-identification location of a re-identified object located at P2(x2, y2) at a time corresponding to a past frame be corrected based on the ROI setting information for region of interest #2.

[0140]

[0141] FIG. 10 schematically illustrates the process of performing the final position derivation step (S500) according to one embodiment of the present invention.

[0142]

[0143] As illustrated in FIG. 10, the final position derivation step (S500) comprises: a positional area generation step for the re-identified object, which generates a positional area of ​​the re-identified object within the area of ​​interest based on the position information of the object tracked in the past frame and the intersection angle; and a final position determination step for determining the position coordinate within the boundary of the positional area closest to the re-identified position or the correction position as the final position when the position coordinate of the re-identified position or the correction position is outside the range of the positional area; wherein the final position is determined as the position coordinate where the shortest distance between the re-identified position or the correction position and the boundary of the positional area is formed.

[0144]

[0145] Schematically, FIG. 10 (a) illustrates an embodiment in which the correction position of the re-identified second object is outside the range of the positionable area, and FIG. 10 (b) illustrates an embodiment in which the correction position of the re-identified second object is not outside the range of the positionable area.

[0146]

[0147] Specifically, the correction position derived by the correction position derivation step (S400) is located within the area of ​​interest, and typically, the vehicle object located on the road moves in correspondence with the direction of travel of the road and does not attempt a sudden lane change within a very short period of time.

[0148] Therefore, the present invention generates the positionable area described in FIG. 7 as the maximum area in which an object on a road can normally move, and if the movement of the object over time deviates from the positionable area, it determines that the detection of the position coordinates of the object was not performed normally and performs correction.

[0149]

[0150] As described above in FIG. 7, the final location derivation step (S500) includes a locationable area generation step and a final location determination step, and the computing system (1000) generates a locationable area of ​​the re-identified object within the area of ​​interest based on the location information of the object tracked in the past frame and the intersection angle for the object re-identified through the locationable area generation step.

[0151] Preferably, the position coordinates of a re-identified object in the current frame are highly likely to be located within a positional area generated based on the intersection angle with the coordinates where the object was located in a past frame. Therefore, the present invention requires the correction position to be corrected once more based on the positional area.

[0152]

[0153] As illustrated in FIG. 10 (a), the corrected position of a re-identified object (C1') located within the region of interest is (a1', b1'), and the coordinates where the re-identified object was located in the past frame are P1(x1, y1). When the corrected position is outside the range (S1) of a positional area created based on P1(x1, y1), the computing system (1000) determines and derives the position coordinate C1(a1", b1"), which is the position coordinate within the boundary line closest to the corrected position in the positional area, where the shortest distance between the corrected position and the boundary line is formed, as the final position of the re-identified object through the final position determination step.

[0154]

[0155] Meanwhile, according to an embodiment, the present invention may derive a final position by correcting the re-identification position of a re-identified object based on the intersection angle through a final position derivation step (S500) without performing a correction position derivation step (S400) that corrects the re-identification position of the re-identified object based on ROI setting information.

[0156] In addition, as in one embodiment illustrated in Fig. 10 (b), when the correction position is located within the positionable area, it is preferable for the final position determination step to derive the position coordinates of the correction position as the final position.

[0157]

[0158] Through the aforementioned series of execution processes, the present invention can achieve the effect of improving the accuracy of multi-object tracking by performing deep learning-based multi-object tracking that considers the environmental information (region of interest) and directional information (cross angle) of an object.

[0159]

[0160] FIG. 11 illustrates the internal configuration of a computing device (11000) according to one embodiment of the present invention.

[0161]

[0162] The computing system (1000) mentioned in the description of FIG. 1 may include components of the computing device (11000) illustrated in FIG. 11, which will be described later.

[0163]

[0164] As illustrated in FIG. 11, the computing device (11000) may include at least one processor (11100), memory (11200), peripheral interface (11300), input / output subsystem (I / O subsystem) (11400), power circuit (11500), and communication circuit (11600).

[0165]

[0166] Specifically, the memory (11200) may include, for example, high-speed random access memory, magnetic disk, SRAM, DRAM, ROM, flash memory, or non-volatile memory. The memory (11200) may include software modules, instruction sets, or various other data required for the operation of the computing device (11000).

[0167] At this time, access to the memory (11200) from other components, such as the processor (11100) or the peripheral device interface (11300), can be controlled by the processor (11100). The processor (11100) may be composed of a single or multiple units and may include processors in the form of GPUs and TPUs to improve computational processing speed.

[0168] The above peripheral device interface (11300) can connect input and / or output peripheral devices of the computing device (11000) to the processor (11100) and the memory (11200). The processor (11100) can perform various functions for the computing device (11000) and process data by executing a software module or instruction set stored in the memory (11200).

[0169] The input / output subsystem (11400) may connect various input / output peripheral devices to the peripheral device interface (11300). For example, the input / output subsystem (11400) may include a controller for connecting peripheral devices such as a monitor, keyboard, mouse, printer, or, if necessary, a touchscreen or sensor to the peripheral device interface (11300). According to another aspect, the input / output peripheral devices may be connected to the peripheral device interface (11300) without passing through the input / output subsystem (11400).

[0170] The power circuit (11500) may supply power to all or part of the components of the terminal. For example, the power circuit (11500) may include one or more power sources such as a power management system, a battery or alternating current (AC), a charging system, a power failure detection circuit, a power converter or inverter, a power status indicator, or any other components for power generation, management, and distribution.

[0171] The communication circuit (11600) may enable communication with another computing device using at least one external port. Alternatively, as described above, the communication circuit (11600) may enable communication with another computing device by including an RF circuit and transmitting and receiving an RF signal, also known as an electromagnetic signal, as needed.

[0172]

[0173] The embodiment of FIG. 11 is merely an example of the computing device (11000), and the computing device (11000) may have some components shown in FIG. 11 omitted, additional components not shown in FIG. 11 added, or a configuration or arrangement that combines two or more components. For example, a computing device for a communication terminal in a mobile environment may include a touchscreen or sensors in addition to the components shown in FIG. 11, and the communication circuit (1160) may include a circuit for RF communication of various communication methods (Wi-Fi, 3G, LTE, 5G, 6G, Bluetooth, NFC, Zigbee, etc.). The components that can be included in the computing device (11000) may be implemented as hardware, software, or a combination of both hardware and software, including one or more integrated circuits specialized for signal processing or applications.

[0174] Methods according to embodiments of the present invention may be implemented in the form of program instructions that can be executed through various computing devices and recorded on a computer-readable medium. In particular, the program according to the present embodiment may be configured as a PC-based program or an application dedicated to a mobile terminal. An application to which the present invention is applied may be installed on a user terminal through a file provided by a file distribution system. For example, the file distribution system may include a file transmission unit (not shown) that transmits the file upon a request from the user terminal.

[0175]

[0176] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0177] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave in order to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be standardized and stored or executed in a standardized manner on a networked computing device. Software and data may be stored on one or more computer-readable recording media.

[0178] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0179]

[0180] In one embodiment of the present invention, by using mathematical algorithms to re-identify objects whose tracking has been interrupted for a plurality of objects detected in each of the past frame and the current frame, the effect of improving the tracking accuracy of multiple objects captured in the image can be achieved.

[0181] In one embodiment of the present invention, by deriving the maximum direction in which an object located within a road can move based on a region of interest set on the road, the tracking accuracy of the movement of a re-identified object can be improved.

[0182] In one embodiment of the present invention, by correcting the position coordinates of an object moving within a road based on a region of interest and an intersection angle generated with respect to the road, the tracking accuracy of the movement of a re-identified object can be improved.

[0183] In one embodiment of the present invention, by generating a positional area where the object can move based on the previous time-based location of the re-identified object, the effect of improving the tracking accuracy of the movement of the re-identified object can be achieved.

[0184] In one embodiment of the present invention, when two or more roads with different directions of travel are captured in an image, the position coordinates at the current time point are corrected by considering the region of interest generated from the road corresponding to the position coordinates at the past time point of the re-identified object and the intersection angle, thereby improving the tracking accuracy of the movement of the re-identified object.

[0185]

[0186] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents. Therefore, other implementations, other embodiments, and equivalents to the claims below are also within the scope of the claims.

Claims

A deep learning-based multi-object tracking method that considers directional information of an object, performed on a computing system comprising 1.1 or more processors and 1 or more memory, wherein A region of interest generation step for generating a region of interest having at least four vertices on the said road according to Region of Interest (ROI) setting information directly set by a person based on video captured on the road; A step for deriving an intersection angle formed by the intersection of diagonals connecting multiple vertices included in the aforementioned region of interest; An object tracking step for deriving tracking information of an object captured in the above video; A correction position derivation step for deriving a corrected position of an object re-identified in the object tracking step by correcting the re-identified position of the object based on the above ROI setting information; and A final position derivation step for deriving the final position of the re-identified object by correcting the re-identification position or correction position of the re-identified object in the object tracking step based on the above intersection angle; A multi-object tracking method in which the above intersection angle represents the maximum directionality that determines the path an object can move on the road.

2. In Claim 1, The above object tracking step is, An object derivation step of deriving first position information of each of a plurality of first objects included in a past frame prior to the current frame of the above image and second position information of each of a plurality of second objects included in the current frame through a deep learning-based detection model; A prediction position derivation step for each of the plurality of first objects, wherein a predicted position at a time corresponding to the current frame is derived through a preset prediction algorithm based on the first position information; An IOU calculation step for calculating an IOU (Intersection Over Union) value for a bounding box for the predicted position of each of the plurality of first objects, and a bounding box for the current position included in the second position information of each of the plurality of second objects; and A multi-object tracking method comprising: a re-identification location derivation step of deriving tracking information including the re-identification location of a second object re-identified as the same object as a first object by applying a plurality of IOU values ​​to a Hungarian algorithm.

3. In Claim 2, The above predicted position derivation step is, For each of the plurality of first objects, an information input step of applying coordinate information, velocity information, and the time difference value between the current frame and the past frame included in the first position information to the prediction algorithm; and A multi-object tracking method comprising: a first object position derivation step for each of the plurality of first objects, wherein the position coordinates at a time corresponding to the current frame predicted by the prediction algorithm are derived as the predicted position.

4. In Claim 2, The above re-identification location derivation step is, A re-identified object extraction step for extracting a second object that is re-identified by applying the above plurality of IOU values ​​to a Hungarian algorithm and determining that it is identical to the first object; An object re-identification step of mapping the ID (Identifier) ​​of the second object being re-identified above to the corresponding ID of the first object; and A multi-object tracking method comprising: a re-identification location determination step for determining the current location derived for the second object being re-identified as the re-identification location.

5. In Claim 1, A multi-object tracking method wherein the above ROI setting information includes the coordinate positions of each of at least four vertices forming the boundary of the region of interest, and an analysis target within the region of interest.

6. In Claim 1, The above intersection angle derivation step is, A diagonal generation step for generating a diagonal connecting opposite vertices among a plurality of vertices included in the region of interest; and A multi-object tracking method comprising: a cross angle selection step for selecting and deriving a cross angle corresponding to the direction of travel of the road among a plurality of cross angles formed by the intersection of a plurality of generated diagonals.

7. In Claim 1, The above correction position derivation step is, If the location coordinates of the above re-identification location fall outside the range of the above region of interest, the location coordinates within the boundary of the above region of interest that are closest to the above re-identification location are derived as the above correction location, and A multi-object tracking method in which the correction position is determined as a position coordinate where the shortest distance between the re-identification position and the boundary of the region of interest is formed.

8. In Claim 1, The above final position derivation step is, A positionable area generation step for the re-identified object, which generates a positionable area of ​​the re-identified object within the interest region based on the position information of the object tracked in the past frame and the intersection angle; and If the position coordinates of the re-identified position or corrected position fall outside the range of the positionable area, the method includes a final position determination step in which the position coordinates within the boundary of the positionable area closest to the re-identified position or corrected position are determined as the final position; A multi-object tracking method in which the above final position is determined by position coordinates where the shortest distance between the above re-identification position or correction position and the boundary of the above positionable area is formed. A computing system comprising a processor of version 9.1 or higher and 1 or more memory, and performing a deep learning-based multi-object tracking method that considers the directional information of objects, Region of Interest generation unit that generates a region of interest having at least four vertices on the said road according to Region of Interest (ROI) setting information directly set by a person based on video captured on the road; A cross-angle derivation unit that derives a cross-angle formed by the intersection of diagonals connecting multiple vertices included in the aforementioned region of interest; An object tracking unit that derives tracking information of an object captured in the above video; A correction position derivation unit that, based on the above ROI setting information, corrects the re-identification position of an object re-identified by the object tracking unit and derives the correction position of the re-identified object; and A final position derivation unit that derives the final position of the re-identified object by correcting the re-identification position or correction position of the re-identified object in the object tracking unit based on the above intersection angle; A computing system in which the above intersection angle represents the maximum directionality that determines the path an object can move on the road.

Images