Operation method of crowd analysis system

Abstract

Disclosed is an operation method of a crowd analysis system. The operation method comprises: a step in which an RGB image-based object detection model performs object detection on an RGB image of a CCTV video; a step for determining whether to segment the RGB image according to a comparison result between the reliability of the object detection result for the RGB image and a preset reference reliability; a step for segmenting the RGB image into a plurality of sub-RGB images upon determining that the RGB image is to be segmented; a step in which a depth image generation model generates a plurality of depth images respectively corresponding to the plurality of sub-RGB images; and a step in which a depth image-based object detection model performs object detection on each of the plurality of depth images, and integrates the object detection results for each depth image to generate object detection results for the plurality of depth images.

Description

How a crowd analysis system works

[0001] The present invention relates to a crowd analysis system, and more specifically, to a method of operation of a crowd analysis system for effectively detecting crowd objects through CCTV footage in a densely populated environment.

[0002] Conventional crowd analysis systems utilizing CCTV footage primarily detect objects using RGB images. However, this approach has limitations when objects are obscured by others or when it is difficult to accurately detect specific objects in dense crowd environments. For instance, RGB images lack depth information, making it difficult to clearly determine the positions of objects. Consequently, the performance of RGB-based object detection technology can significantly degrade in dense environments.

[0003] Recently, technology is being developed to analyze RGB and depth images together by installing depth cameras in addition to CCTV systems. Such technologies have the advantage of providing various services with high accuracy that conventional RGB cameras could not offer. However, many CCTV systems still use RGB cameras exclusively, and the additional installation of depth cameras is limited due to issues such as battery consumption and equipment costs. Consequently, existing CCTV systems using only RGB cameras face limitations in utilizing depth image information.

[0004] The present invention, aimed at solving the aforementioned problems, is intended to provide a method of operation for a crowd analysis system using CCTV video, which converts RGB images of CCTV video into depth images and, based thereon, accurately detects crowd objects (human objects) in real time in a crowded environment.

[0005] A method of operation of a crowd analysis system according to the present invention for achieving the above-mentioned purpose comprises: a step in which an RGB image-based object detection model performs object detection on an RGB image of a CCTV video; a step in which a determination is made whether to segment the RGB image based on a comparison result between the reliability of the object detection result for the RGB image and a pre-set reference reliability; a step in which, if the segmentation of the RGB image is determined, the RGB image is segmented into a plurality of sub-RGB images; a step in which a depth image generation model generates a plurality of depth images corresponding to each of the plurality of sub-RGB images; and a step in which a depth image-based object detection model performs object detection on each of the plurality of depth images and integrates the object detection results for each depth image to generate an object detection result for the plurality of depth images.

[0006] According to the present invention, by dividing an RGB image of a CCTV video into a plurality of sub-RGB images and performing object detection on a plurality of depth images corresponding to each of the plurality of sub-RGB images, it is possible to reduce battery consumption and installation costs associated with the additional installation of a depth camera in addition to an RGB camera in an existing CCTV system, and to expect improved object detection performance.

[0007] In addition, the RGB image-based object detection model can be automatically optimized by fine-tuning the parameters (weights and biases) of the RGB image-based object detection model based on the object detection results for multiple depth images.

[0008] Figure 1 is a configuration diagram of a crowd analysis system according to an embodiment of the present invention.

[0009] Figure 2 is a flowchart illustrating the operation method of the crowd analysis system of Figure 1.

[0010] Figure 3 is a diagram schematically illustrating the process of the depth image generation model of Figure 1 converting the entire RGB image into a depth image.

[0011] Figure 4 is a diagram schematically illustrating the process of the depth image generation model of Figure 1 converting a sub-RGB image separated from an RGB image into a depth image.

[0012] FIG. 5 is a diagram schematically illustrating the process of integrating object detection results for multiple depth images converted from multiple sub-RGB images according to an embodiment of the present invention.

[0013] Figure 6 is a diagram illustrating the update process of an RGB image-based object detection model by the parameter tuning module of Figure 1.

[0014] The terms used in this specification are used merely to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, terms such as “comprising” or “having” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0015] A crowd analysis system according to an embodiment of the present invention can improve the accuracy of object detection by converting an RGB image obtained from a CCTV video into a depth image and performing object detection based on the converted depth image.

[0016] In addition, the crowd analysis system according to an embodiment of the present invention can further increase the accuracy of object detection by appropriately dividing the RGB image into sub-RGB images during the process of converting the RGB image into a depth image, and individually performing object detection on the depth images corresponding to the divided sub-RGB images.

[0017] In addition, the crowd analysis system according to an embodiment of the present invention can optimize an existing RGB image-based object detection model by fine-tuning the existing RGB image-based object detection model using object detection results for depth images.

[0018] Hereinafter, a crowd analysis system (100) according to an embodiment of the present invention will be described in detail with reference to the drawings.

[0019] FIG. 1 is a configuration diagram of a crowd analysis system according to an embodiment of the present invention, and FIG. 2 is a flowchart showing the operation method of the crowd analysis system of FIG. 1.

[0020] Referring to FIGS. 1 and 2, in step S110, an RGB image-based object detection model (110) performs object detection on the RGB images of the CCTV video. In this embodiment, the object to be detected may be a crowd object or a person object. In this embodiment, object detection refers to a process of detecting objects based on RGB images acquired frame by frame from the input CCTV video. For example, object detection may involve detecting a bounding box surrounding an object. Here, the bounding box may refer to a rectangular area surrounding the object. The bounding box may be defined, for example, by the coordinates (x, y) of the top-left corner of the rectangular area, the width (w), and the height (h). Additionally, object detection may involve inferring the probability that an object within the detected bounding box belongs to the person class. In this embodiment, this probability is referred to as the person class prediction probability (R). i It shall be referred to as ). Therefore, the object detection result obtained by an RGB image-based object detection model performing object detection on an RGB image comprises data related to bounding boxes and human class prediction probabilities (R i ...includes ). Here, the data related to the bounding box includes, for example, the size of the bounding box (x, y, w, h), the total number of bounding boxes, and the ratio of the size of the bounding box to the total size of the RGB image (W i It includes ). A deep learning model can be used as an RGB image-based object detection model (110) that performs such object detection, and examples of such deep learning models include YOLO (You Only Look Once), Faster R-CNN (region-based convolutional neural networks), SSD (Single Shot MultiBox Detector), etc.

[0021]

[0022] Next, in step S120, the Image Segmentation Decision Module (120) determines whether to segment the RGB image based on the result of comparing the reliability of the object detection result for the RGB image with a pre-set reference reliability. The Image Segmentation Decision Module (120) includes a reliability calculation module (122) and a comparison module (124).

[0023] The step of determining whether to segment an RGB image (S120) includes the step of calculating the reliability of the object detection result for the RGB image in the reliability calculation unit (122) and the step of comparing the reliability with the reference reliability in the comparison unit (124).

[0024] The step of calculating the reliability is a step of evaluating the object detection results for the RGB image. In one embodiment, the reliability may be calculated based on the bounding box size and the human class prediction probability.

[0025] More specifically, the step of calculating the reliability comprises the total number of detected bounding boxes (M) included in the object detection result, and the person class prediction probability (R) representing the probability that an object within the bounding box belongs to the person class. i The step may be to calculate the reliability using the ratio (Wi) of the bounding box size to the total size of the RGB image.

[0026] The above reliability calculation may utilize a weighted average reliability calculation method that reflects higher reliability as the size of each bounding box area decreases, and the reliability may be calculated based on detection results for small objects. The above reliability can be calculated, for example, by the following Equation 1.

[0027]

[0028]

[0029] M: Total number of detected bounding boxes

[0030] R i : Person class prediction probability of bounding box i

[0031] W i : Ratio of the area size of bounding box i to the total image (RGB image) (Size ratio = Bounding box size / Total image size)

[0032] α i : Indicator function for critical probability filtering

[0033] R i α when at the threshold i =1, otherwise α i =w, where w is an arbitrary weight

[0034] According to the above mathematical formula 1, the smaller the proportion of the bounding box in the entire image, the higher the confidence is assigned, and the confidence of each bounding box is calculated based on the human class prediction probability. In addition, to reflect the prediction uncertainty of small objects, a weight (w) is assigned to object detection results below a certain threshold (certain threshold probability).

[0035]

[0036] Next, in step S130, if the division of the RGB image is determined by the image division determination unit (120), the image division unit (130) divides the RGB image into a plurality of sub-RGB images.

[0037] The case in which the segmentation of the RGB image is determined may, for example, when the comparison unit (124) in step S120 compares the reliability with the reference reliability and the result shows that the reliability is smaller than the reference reliability. Accordingly, if the reliability is smaller than the reference reliability, the image segmentation determination unit (120) divides the RGB image into a plurality of sub-RGB images. If the reliability is greater than or equal to the reference reliability, the steps (S130 to S160) described below are not performed and the process terminates.

[0038] In an embodiment, the step (S130) of dividing the RGB image into a plurality of sub-RGB images includes the step of generating a division method (A) for optimizing the division of the RGB image in a division method generation model (160) and the step of dividing the RGB image into a plurality of sub-RGB images according to the division method (A) in an image division unit (130).

[0039] In an embodiment, the segmentation method (A) may be data or information related to a segmentation strategy for determining the number of the plurality of sub-RGB images and the size of each sub-RGB image. The segmentation method generation model (160) that generates the segmentation method (A) may be, for example, a reinforcement learning model that generates the segmentation method (A).

[0040] In an embodiment, step S130 may further include a step in which a segmentation method generation model (160) learns a segmentation method (A).

[0041] In an embodiment, the step of learning the segmentation method (A) may include adjusting the parameters of the segmentation method generating model based on the number (K) of the plurality of sub-RGB images representing the segmentation state (S) of the RGB images and the reward value (R) included in the object detection result for the plurality of depth images. Here, the parameters may include weights and biases. The weights are adjusted during the learning process of the segmentation method (A) and may be values ​​representing the importance of the input data (S, R) input to the segmentation method generating model (160) when it is transmitted to each neuron of the neural network constituting the segmentation method generating model (160). The bias may be an additional parameter that adjusts the activation function of the neuron.

[0042] The segmentation state (S) of an RGB image can be defined as a set of currently segmented regions. A segmentation method generation model (160) is a specific segmentation state (S) of an RGB image. t The optimal partitioning method (A) in ) t Generates ). The reward value (R) represents the reward based on the degree of improvement in object detection performance due to image segmentation.

[0043] The partitioned state (S) is 'S = R1, … , R K It can be defined as follows. K is defined as the number of partitioned regions. R K is 'R K = (x k,min , x k, max )×(y k,min , y k, max It can be defined as )'. The partitioning method (A) is the addition of a region (A t,add ), area adjustment (A t,adjust ) and area merging (A t,merge It includes ) and can be expressed by the following formula 2.

[0044]

[0045]

[0046]

[0047] In the embodiment, the compensation value (R) is a compensation value (R) according to the object detection growth rate. increase Compensation value (R) based on ) and object density density It can be a combination of ).

[0048] In the embodiment, the compensation value (R) according to the object detection increase rate is increase ) is the segmentation state (S) of the RGB image previously performed. t-1 Number of objects detected in ) (N t-1 ) and the segmentation state of the above RGB image performed at present (S t Number of objects detected in ) (N t It may be a value calculated based on ). In a specific embodiment, the compensation value (R) according to the object detection increase rate. increase ) is the segmentation state (S) of the RGB image previously performed. t-1 Number of objects detected in ) (N t-1 ) and the segmentation state of the above RGB image performed at present (S t Number of objects detected in ) (N t The sum of ) is the segmentation state (S) of the RGB image performed previously. t-1 Number of objects detected in ) (N t-1 It can be a value divided by ).

[0049] In the embodiment, the compensation value (R) according to the object density density ) is the segmentation state of the above RGB image performed at present (S t Number of objects detected in ) (N t The value obtained by summing the sizes of each of the aforementioned multiple sub-RGB images ( It may be a value calculated based on ). In a specific embodiment, the compensation value (R) according to the object density density ) is the segmentation state (S) of the above RGB image performed at present. t Number of objects detected in ) (N t The value obtained by summing the sizes of each of the aforementioned multiple sub-RGB images ( It can be a value divided by ).

[0050] The reward value (R) based on this object detection growth rate increase Compensation value (R) based on ) and object density density ) can be specifically expressed by the following Equation 3.

[0051]

[0052]

[0053] Next, in step S140, a depth image generation model (140) generates a plurality of depth images corresponding to each of the plurality of sub-RGB images that are separated from the RGB image by the image segmentation unit (130). The depth image generation model (140) may be, for example, a deep learning model pre-trained to convert an RGB image into a depth image. Examples of such deep learning models include Convolutional Neural Networks (CNNs) models, Transformers-based deep learning models, Recurrent Neural Networks (RNNs) models, Generative Adversarial Networks (GANs) models, and Autoencoders.

[0054] FIG. 3 is a diagram schematically illustrating the process of a depth image generation model according to an embodiment of the present invention converting an entire RGB image into a depth image, and FIG. 4 is a diagram schematically illustrating the process of a depth image generation model according to an embodiment of the present invention converting a sub-RGB image divided from an RGB image into a depth image.

[0055] As shown in FIG. 3, rather than converting the entire RGB image into a depth image and performing object detection on the depth image, performing object detection individually on multiple sub-RGB images divided from the RGB image as shown in FIG. 4 can estimate depth more accurately.

[0056] When converting a full RGB image into a depth image, details of individual objects may be diluted because various object and background information must be processed simultaneously across a wide field of view. However, if the image is divided into sub-RGB images and processed individually, finer information about the objects and their surrounding environments contained in the depth images corresponding to each sub-RGB image is preserved, which can lead to more accurate depth estimation of individual objects. In other words, while complex backgrounds can negatively affect depth estimation when performing object detection on depth images corresponding to a full RGB image, processing the image by dividing it into sub-RGB images allows the depth images corresponding to each sub-RGB image to contain relatively simple backgrounds, thereby improving the accuracy of object detection and depth estimation.

[0057]

[0058] Next, in step S150, a depth image-based object detection model (150) performs object detection for each of the multiple depth images. Additionally, the depth image-based object detection model (150) can generate object detection results for the multiple depth images (41, 42, 43, and 44 in FIG. 5) by integrating the object detection results for each depth image, as illustrated in FIG. 5. The depth image-based object detection model (150) may be a deep learning model pre-trained to perform object detection in depth images. Examples of such deep learning models include Faster R-CNN with Depth, Single Shot Detector (SSD) based models, YOLO based models, Transformer based models, GANs based models, etc.

[0059]

[0060] Additionally, in step S160, the parameter tuning module (170) can update the RGB image-based object detection model (110) using the object detection results for a number of depth images (41, 42, 43 and 44 of FIG. 5).

[0061] Figure 6 is a diagram illustrating the update process of an RGB image-based object detection model by the parameter tuning module of Figure 1.

[0062] Referring to FIG. 6, the update step of the RGB image-based object detection model (110) by the parameter tuning module (170) may be a step of fine-tuning the RGB image-based object detection model (110) by reflecting the object detection result (e.g., feature vector, etc.) inferred from the depth image-based object detection model (150) into the RGB image-based object detection model (110).

[0063] An RGB image-based object detection model (110) and a depth image-based object detection model (150) each use multiple equally divided sub-RGB images and multiple depth images as inputs to output respective object detection results (e.g., feature vectors generated in each last layer). A parameter tuning module (170) measures the difference between the two output object detection results (feature vectors) and reflects this in a loss function to fine-tune the RGB image-based object detection model using an optimization algorithm. Here, for example, Gradient Descent may be used as the optimization algorithm.

[0064] More specifically, the update step of the RGB image-based object detection model (110) by the parameter tuning module (170) includes the step of performing object detection on a plurality of sub-RGB images in the RGB image-based object detection model (110), integrating the object detection results for the plurality of sub-RGB images to generate an object detection result obtained, and the step of updating the RGB image-based object detection model (110) by fine-tuning the parameters (weights and biases) of the RGB image-based object detection model (110) in a direction that minimizes the loss function representing the difference between the object detection result for the plurality of sub-RGB images and the object detection result for the plurality of depth images in the parameter tuning module (170).

[0065]

[0066] Meanwhile, the crowd analysis system (100) illustrated in FIG. 1 may further include a processor (181), memory (182), storage device (183), and communication device (184).

[0067] The processor (181) is a device that controls the operation and / or execution of the aforementioned configurations (110 to 170) and may include at least one CPU (Central Processing Unit) core, at least one GPU (Graphics Processing Unit), and at least one NPU (Neural Processing Unit) divided into tiles.

[0068] The memory (182) may be a device that temporarily stores data necessary for the operation and / or execution of the aforementioned configurations (110–170) and provides it to the processor (181) upon request. The memory (182) may be included within the processor (181).

[0069] The storage device (183) may be a non-volatile storage device that stores a software module when the aforementioned configurations (110 to 170) are implemented as a software module. Additionally, the storage device (183) may store an operating system that provides an execution environment for the aforementioned configurations (110 to 170).

[0070] The communication device (184) may be a device that supports communication between the crowd analysis system (100) and an external device. The external device may be, for example, a CCTV system including a CCTV camera. Communication includes wired and / or wireless communication. Wireless communication includes Bluetooth, Wi-Fi, mobile communication, etc. The crowd analysis system (100) may be included within the CCTV system.

[0071]

[0072] The embodiments disclosed in this specification should be considered in an exemplary sense for the sake of illustration rather than in a limiting sense. The scope of the invention is defined by the claims, not by the foregoing description, and all variations within the scope of the claims should be interpreted as being included in the invention.

[0073] The crowd analysis system according to the present disclosure has industrial applicability in the field of crowd analysis using CCTV.

Claims

1. A step in which an RGB image-based object detection model performs object detection on the RGB images of a CCTV video; A step of determining whether to segment the RGB image based on the result of comparing the reliability of the object detection result for the RGB image with a pre-set reference reliability; When the division of the above RGB image is determined, a step of dividing the above RGB image into a plurality of sub-RGB images; A depth image generation model generates a plurality of depth images corresponding to each of the plurality of sub-RGB images; and A depth image-based object detection model performs object detection on a plurality of depth images, respectively, and integrates the object detection results for each depth image to generate object detection results for the plurality of depth images. A method of operation of a crowd analysis system including 2. In Paragraph 1, A step of determining whether to segment the above RGB image, A step of calculating the reliability of the object detection result for the above RGB image; and It includes a step of comparing the above reliability with the above standard reliability, The step of calculating the above reliability is, A method of operation of a crowd analysis system, comprising the step of calculating the reliability using the total number of detected bounding boxes included in the object detection result, a person class prediction probability indicating the probability that an object within the bounding box belongs to the person class, and the ratio of the bounding box size to the total size of the RGB image.

3. In Paragraph 1, The step of dividing the above RGB image into a plurality of sub-RGB images is, A method of operation of a crowd analysis system, wherein if the above reliability is smaller than the above reference reliability, the above RGB image is divided into a plurality of sub-RGB images.

4. In Paragraph 1, The step of dividing the above RGB image into a plurality of sub-RGB images is, A segmentation method generation model generates a segmentation method for optimizing the segmentation of the RGB image; and A step of dividing the RGB image into a plurality of sub-RGB images according to the above division method A method of operation of a crowd analysis system including 5. In Paragraph 4, The above division method A method of operation of a crowd analysis system, which is a segmentation strategy for determining the number of multiple sub-RGB images and the size of each sub-RGB image.

6. In Paragraph 4, The above-mentioned segmentation method generation model further includes the step of learning the above-mentioned segmentation method, and The step of learning the above-mentioned division method is, A method of operation of a crowd analysis system, wherein the method involves adjusting the parameters of a segmentation method generation model based on the number of sub-RGB images representing the segmentation state of the RGB image and the reward value included in the object detection result for the multiple depth images to learn the segmentation method.

7. In Paragraph 4, The above compensation value is, A method of operation for a crowd analysis system that is the sum of a reward value based on the object detection growth rate and a reward value based on object density.

8. In Paragraph 7, The compensation value based on the above object detection increase rate is, The number of objects detected in the segmentation state of the RGB image previously performed and the number of objects detected in the segmentation state of the RGB image currently performed (N t A method of operation for a crowd analysis system, which is a value calculated based on ).

9. In Paragraph 7, The compensation value based on the above object detection increase rate is, A method of operation of a crowd analysis system, wherein the sum of the number of objects detected in the previously performed segmentation state of the RGB image and the number of objects detected in the current segmentation state of the RGB image is divided by the number of objects detected in the previously performed segmentation state of the RGB image.

10. In Paragraph 7, The compensation value based on the above object density is, A method of operation of a crowd analysis system, which is a value calculated based on the sum of the number of objects detected in the segmentation state of the RGB image performed at present and the size of each of the multiple segmented sub-RGB images.

11. In Paragraph 7, The compensation value based on the above object density is, A method of operation of a crowd analysis system, wherein the number of objects detected in the segmentation state of the RGB image performed at present is divided by the sum of the sizes of each of the multiple segmented sub-RGB images.

12. In Paragraph 1, After the step of generating the integrated object detection results mentioned above, The above RGB image-based object detection model performs object detection on each of the plurality of sub-RGB images and generates an object detection result obtained by integrating the object detection results for the plurality of sub-RGB images; and A parameter tuning module updates the RGB image-based object detection model by fine-tuning the parameters of the RGB image-based object detection model in a direction that minimizes a loss function representing the difference between the object detection results for the plurality of sub-RGB images and the object detection results for the plurality of depth images. A method of operation for a crowd analysis system that further includes

Images