Image pre-processing system and method for reducing transmission load in intelligent CCTV networks

Abstract

Disclosed is an image pre-processing system including an intelligent CCTV and a server, wherein the intelligent CCTV comprises: a camera which captures an image; a reference image generator which models a background image by accumulating images; a DSSIM-MAP generator which constructs a DSSIM-MAP by comparing a current image captured by the camera with the background image on an area-by-area basis; an ROI filter which removes color except for an ROI on the basis of the DSSIM-MAP; and a communication unit which transmits, to the server, the image from which the color has been removed except for the ROI, wherein the server comprises: a communication unit which receives the image from the intelligent CCTV; and an object detector which detects an object by filling an area from which the color has been removed in the received image with a predetermined color.

Description

Image preprocessing system and method for reducing transmission load in an intelligent CCTV network environment

[0001] The present invention relates to an image preprocessing system and method for reducing transmission load in a network environment of an intelligent CCTV, and more specifically, to a technology for preprocessing images to transmit only the data necessary for object detection in order to reduce the transmission load in the intelligent CCTV and the processing load in the server.

[0002] Intelligent CCTVs can identify objects by analyzing recorded video using artificial intelligence technology. Since the AI ​​used generally utilizes convolution-based deep learning, it typically involves a high volume of computation. Because deploying deep learning hardware to every individual CCTV is a significant financial burden and difficult to manage, intelligent CCTVs transmit video footage to a server for processing. However, recent CCTV footage is high-resolution, and streaming all of it places a heavy load on the network. Furthermore, the server that aggregates this footage and processes it using deep learning must also bear a heavy processing load. These network load factors can reduce the practicality of expanding or operating CCTV systems.

[0003] Accordingly, a series of techniques are required to perform preprocessing that removes unnecessary image content for deep learning so that compression algorithms performed before data transmission can operate more efficiently, and for the server to ensure that the accuracy of the received image does not decrease compared to the original image in the convolutional neural network.

[0004] The present invention aims to provide an image preprocessing system and method for reducing transmission load in a network environment of an intelligent CCTV.

[0005] As a technical means for achieving the above-mentioned objective, one embodiment of the present invention may provide an image preprocessing system comprising an intelligent CCTV and a server, wherein the intelligent CCTV comprises a camera for capturing images, a reference image generator for accumulating images to model a background image, a DSSIM-MAP generator for constructing a DSSIM-MAP by comparing the current image captured by the camera with the background image by region, an ROI filter for removing colors excluding ROIs based on the DSSIM-MAP, and a communication unit for transmitting the image with colors removed excluding ROIs to a server, and the server comprises a communication unit for receiving images from the intelligent CCTV and an object detector for detecting objects by filling a predetermined color into the color-removed area of ​​the received image.

[0006] In addition, the reference image generator can generate background images using Gaussian mixture modeling.

[0007] In addition, the DSSIM-MAP for measuring the dissimilarity between the background image and the current image has Equation 1, and Equation 1 is (where x is the current image, y is the background image, is the average, is the standard deviation, and is a constant) can be.

[0008] Additionally, the ROI filter sets a threshold based on the DSSIM-MAP, determining areas exceeding the threshold as ROIs and areas below the threshold as non-ROIs; non-ROIs are replaced with a solid color, while ROI areas are represented in their original colors. Furthermore, to reduce the file size of the image transmitted to the server, non-ROIs in the ROI filter are replaced with white or black, which have a higher compression ratio. The object detector can fill the areas from which color has been removed with colors of higher saturation to further improve the object recognition rate.

[0009] In addition, the system is intended to track vehicle information using a CCTV network, the intelligent CCTV is installed on the road, and the object may include a vehicle.

[0010] In addition, intelligent CCTVs can use run-length encoding to reduce the volume of video transmitted to the server.

[0011] In addition, the reference image generator can learn criteria for generating background images from multiple images captured by a camera.

[0012] It has the advantage of allowing the operation of more CCTVs by reducing the transmission load of the network between the CCTV and the server.

[0013] It also reduces the processing load on the server side for object detection, providing the advantageous effect of effectively implementing and intelligentizing CCTV networks compared to existing methods.

[0014] By removing unnecessary image content for deep learning, the compression algorithm performed before data transmission can operate more efficiently, and there is a technical effect of ensuring that the accuracy of the received image does not decrease compared to the original image in the convolutional neural network.

[0015] FIG. 1 is a diagram schematically illustrating an image preprocessing system according to one embodiment.

[0016] FIG. 2 is a flowchart illustrating an image preprocessing method according to one embodiment.

[0017] FIG. 3 is a drawing showing an example of a background image according to one embodiment.

[0018] FIG. 4 is a diagram showing the result of a DSSIM-MAP generator according to one embodiment.

[0019] Figure 5 is a diagram showing the result of an ROI filter according to one embodiment.

[0020] Figure 6 is a graph showing the transmission bit measurement results according to a threshold value according to one embodiment.

[0021] FIG. 7 is a drawing showing an example in which a predetermined color is filled into an area from which color has been removed according to one embodiment.

[0022] FIG. 8 is a diagram showing the results of measuring inference time according to DSSIM and RPN threshold values ​​according to one embodiment.

[0023] The invention will be described in detail below by way of exemplary embodiments with reference to the attached drawings. The following embodiments are intended only to embody the invention and do not limit or restrict the scope of the invention. Anything that can be easily inferred by a person skilled in the art to which the invention pertains from the detailed description and embodiments shall be interpreted as falling within the scope of the invention.

[0024] Terms such as 'composed' or 'comprising' as used in this specification should not be interpreted as necessarily including all of the various components or steps, and should be interpreted as some of the components or steps may not be included, or additional components or steps may be included.

[0025] The terms used in this specification are described in their current, general form considering the functions mentioned herein; however, they may refer to various other terms depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Therefore, the terms used in this specification should not be interpreted solely by their names, but should be interpreted based on the meaning of the terms and the overall content of this specification. Furthermore, singular expressions include a plural meaning unless the context clearly indicates a singular meaning.

[0026] The terms “above” and similar designations used in this specification (particularly in the claims) may indicate both singular and plural forms. Furthermore, unless there is a description explicitly specifying the order of the steps describing the method according to this specification, the described steps may be performed in a suitable order. The present invention is not limited by the order in which the described steps are described.

[0027] Phrases such as "in one embodiment" appearing in various places in this specification do not necessarily refer to the same embodiment.

[0028] Some embodiments of this specification may be represented by functional block configurations and various processing steps. Some or all of these functional blocks may be implemented by various numbers of hardware and / or software configurations that execute specific functions.

[0029] The present embodiments relate to an image preprocessing system and method for reducing transmission load in a network environment of an intelligent CCTV, and detailed descriptions of matters widely known to those skilled in the art to which the following embodiments belong are omitted. The present invention will be described in detail below with reference to the attached drawings.

[0030] FIG. 1 is a diagram schematically illustrating an image preprocessing system according to one embodiment. An image preprocessing system (100) according to one embodiment may include an intelligent CCTV (110) and a server (120). However, not all of the illustrated components are essential components. The system (100) may be implemented with more components than illustrated, or with fewer components. The above components will be examined in turn below.

[0031] An intelligent CCTV (110) according to one embodiment may include a camera (111), a reference image generator (113), a DSSIM-MAP generator (115), an ROI filter (117), and a communication unit (not shown).

[0032] In the case of an intelligent CCTV (110), for monitoring and analysis using artificial intelligence, the camera (111) captures video of the scene and transmits it to the server (120) to obtain object detection results. At this time, the intelligent CCTV (110) undergoes a preprocessing process before transmitting the video captured by the camera (111) to the server (120), and during this process, a reference image generator (113), a DSSIM-MAP generator (115), and an ROI filter (117) can operate.

[0033] The reference image generator (113) can model a background image by accumulating images captured by the camera (111). For example, the reference image generator (113) can learn criteria for generating a background image from multiple images captured by the camera (111).

[0034] The DSSIM-MAP generator (115) can construct a DSSIM-MAP by comparing the current image captured by the camera (111) with the background image (e.g., reference image) modeled by the reference image generator (113) in terms of regions. The ROI filter (117) can set a threshold on the DSSIM-MAP to retain color only in the region of interest and remove color from the rest of the area.

[0035] The Structural Similarity Index (SSIM) is a method for predicting the perceived quality of digital television, motion pictures, and other types of digital images and videos. It is also used to measure the similarity between two images. The SSIM index is a global reference metric; that is, the measurement or prediction of image quality is based on an initial uncompressed or distortion-free image as a reference.

[0036] SSIM-MAP maps the similarity of corresponding points in two images using a structural similarity index. Areas with high similarity receive a score of 1.0, while areas with low similarity receive a score of -1.0.

[0037] The DSSIM (Divergence of Structural Similarity Index) map is a method that calculates the structural differences between two images based on the SSIM (Structural Similarity Index) metric. Unlike SSIM, DSSIM can be interpreted as the difference in SSIM values, which calculate the degree of similarity between two images. In other words, DSSIM is an indicator that measures how different and dissimilar two images are.

[0038] In other words, SSIM is a direct metric that calculates the similarity between images and has a value between 0 and 1. A value closer to 1 indicates that the images are more similar. DSSIM is a metric that emphasizes the difference from SSIM, indicating that a higher DSSIM value signifies a greater difference between the images. The DSSIM value ranges from 0 to 1, meaning that a value closer to 0 indicates higher similarity.

[0039] A communication unit (not shown) can transmit and receive information with external devices (e.g., a server). The external device may be a computing device, but is not limited thereto. The communication unit (not shown) may include one or more components that enable the intelligent CCTV (110) to communicate with the server (120). The communication unit of the intelligent CCTV (110) can encode a color-removed image of the area excluding the ROI in the DSSIM-MAP generator (115) and transmit it to the server (120).

[0040] In one embodiment, since bitmap color images occupy a large amount of space when pixel information is represented in its raw state, it is common practice to reduce the size using a compression algorithm for storage or transmission. As an example of a compression algorithm, Run-length encoding encodes and compresses data when the same value or pattern appears consecutively in order to reduce the large amount of space. Accordingly, the present technology can preprocess images by removing content that is unnecessary for performing deep learning, thereby enabling the compression algorithm performed before data transmission to operate more efficiently.

[0041] A server (120) according to one embodiment may include an object detector (121) and a communication unit (not shown).

[0042] A communication unit (not shown) can transmit and receive information with external devices. The external devices may be computing devices, but are not limited thereto. The communication unit (not shown) may include one or more components that enable the intelligent CCTV (110) to communicate with the server (120). The communication unit of the server (120) can receive encoded video data from the intelligent CCTV (110).

[0043] The object detector (121) can perform object detection by filling a specific color into the area removed in the previous process from the image received from the intelligent CCTV (110). For example, the object detector (121) can detect objects using an artificial intelligence model trained for object detection. The purpose of computer vision is to extract meaningful information from images. Object recognition is one of the computer vision techniques for identifying objects in images or videos. That is, it means detecting semantic object instances of a specific class (e.g., people, buildings, or cars) in digital images and videos. At this time, the location of the object can be output as a bounding box or segment.

[0044] Accordingly, the server (120) can receive video data in real time from the intelligent CCTV (110), detect objects through an AI model, and determine the type of object. In addition, the server (120) can immediately provide prediction results.

[0045] Here, the system (100) is for tracking vehicle information using a CCTV network, the intelligent CCTV (110) is installed on the road, and the object may include a vehicle.

[0046] FIG. 2 is a flowchart illustrating an image preprocessing method according to one embodiment.

[0047] In step S210, the camera can capture video. In one embodiment, video can be captured through a camera installed within an intelligent CCTV. For example, the video may include video and still images. For example, video containing vehicle information can be captured through a CCTV installed on a highway.

[0048] In step S220, a background image can be modeled by accumulating images. In one embodiment, images captured by a camera can be accumulated over time and a background image can be generated using Gaussian mixture modeling. That is, criteria for generating a background image from multiple images captured by a camera can be learned.

[0049] A background image can be composed of regions that remain relatively unchanged over long periods, serving as a type of background. Background images are necessary to detect areas of difference where current changes have occurred by comparing them with the current image. While it is difficult to obtain a background completely devoid of objects in a real-world environment, background images can be generated by probabilistically selecting color values ​​with a high frequency of appearance. Gaussian Mixture Modeling can be utilized for this purpose.

[0050] For an example of modeling a background image, we will look into it in more detail with reference to Fig. 3.

[0051] In step S230, a DSSIM-MAP can be constructed by comparing the current video captured by the camera and the background image by region.

[0052] A DSSIM-MAP is a visual representation of the calculated DSSIM value for each pixel of an image, serving as a map that visualizes the differences between two images at the pixel level. Through a DSSIM-MAP, one can intuitively identify which areas of the two images differ the most and which areas are similar.

[0053] To obtain the DSSIM-MAP for measuring the dissimilarity between the background image and the current image, the following formula is used.

[0054]

[0055]

[0056] In mathematical equation 1, x is the current image, y is the background image, is the average, is the standard deviation, and is a constant.

[0057] An example of the result of generating the DSSIM-MAP will be described later with reference to Fig. 4.

[0058] In step S240, colors can be removed excluding the ROI (Region of Interest) based on the DSSIM-MAP.

[0059] The ROI can be determined using an ROI filter, which sets a threshold based on the DSSIM-MAP; areas exceeding the threshold are designated as ROIs, while areas below the threshold are designated as non-ROIs. Accordingly, non-ROIs can be replaced with a solid color, while ROI areas can be displayed in their original colors.

[0060] In other words, the purpose of the ROI filter is to extract the ROI region by thresholding based on the DSSIM-MAP. Areas below the threshold become Non-ROI, while areas above the threshold become ROI. Meanwhile, since the DSSIM-MAP was created by reducing the image size in the previous step, it can be enlarged back to its original size.

[0061] In the original image, non-ROI areas are replaced with a solid color, while ROI areas are represented in their original colors. For example, non-ROI areas can be replaced with white or black, which offer higher compression ratios, to reduce the file size of the image transmitted to the server. Meanwhile, areas processed in this way enhance the compression efficiency of the image codec and have the effect of reducing the amount of data transmitted over a network.

[0062] An example of the result of applying the ROI filter will be described later with reference to Fig. 5.

[0063] In step S250, color-removed video excluding the ROI can be transmitted to the server. To this end, the intelligent CCTV and the server may be connected via a wired or wireless network. The intelligent CCTV may use run-length encoding to reduce the file size of the video transmitted to the server.

[0064] Bitmap color images occupy a large amount of storage space when pixel information is represented in its raw state. Therefore, it is common practice to reduce their size using compression algorithms for storage or transmission. As an example of such an algorithm, Run-length encoding compresses data by encoding consecutive identical values ​​or patterns to reduce large storage capacity.

[0065] In step S260, an object can be detected by filling a color-removed area in the image received from the intelligent CCTV with a predetermined color. For example, the identified object may be a car driving on a road.

[0066] The server can perform object detection by filling the areas removed in the previous process from the received image with a specific color. In one embodiment, to further improve the object recognition rate, a color with higher saturation can be filled into the areas where color was removed. For example, a color with higher saturation can be green, and in RGB, it can be (255,0,0), (0,255,0), or (0,0,255). That is, it can be seen that the intelligent CCTV replaces the non-ROI area with a solid color without saturation, such as white or black, while the server replaces the non-ROI area with a color with high saturation (e.g., green).

[0067] An example of detecting objects by filling in color will be described later with reference to Fig. 7.

[0068] FIG. 3 is a drawing showing an example of a background image according to one embodiment.

[0069] A reference image generator can generate a background image by learning from multiple images captured by a camera. Figure 3 shows a background image generated by the reference image generator, which is generated in black and white. As a type of background, the background image can be composed of areas that change little over a long period of time. That is, looking at Figure 3, it may be an image that includes the road surface and mountains at the edge, excluding vehicles traveling on a highway (e.g., buses, cars, and trucks). The background image is necessary to detect the difference area where a current change has occurred by comparing it with the current image. Although it is difficult to obtain a background completely devoid of objects in a real environment, a background image can be generated by probabilistically selecting color values ​​with a high frequency of background appearance. To this end, Gaussian Mixture Modeling can be utilized.

[0070] FIG. 4 is a diagram showing the result of a DSSIM-MAP generator according to one embodiment.

[0071] DSSIM-MAP measures the dissimilarity between a reference image and a camera image by placing a fixed window at each point in the image. When using simple pixel comparison as a method to measure dissimilarity, there is a problem in that it is difficult to detect objects when the background and the object have similar colors. Additionally, in the case of cameras installed outdoors, applying simple pixel comparison can result in a sensitive response when shaking occurs.

[0072] As an alternative to this, the present specification may use a DSSIM-MAP based on a SSIM-MAP. The DSSIM-MAP is based on the SSIM formula, which is calculated using the following formula.

[0073]

[0074]

[0075]

[0076] SSIM is an equation regarding similarity that can be calculated using the region mean, region variance, and a constant. The value ranges from -1.0 to 1.0, with higher values ​​indicating higher similarity.

[0077] Transforming this into an expression for dissimilarity, it can be summarized in the following formula.

[0078]

[0079]

[0080]

[0081] DSSIM has a value between 0 and 1.0, with higher values ​​indicating a greater difference from the original. To create a DSSIM-MAP, the original image and the current camera image are scaled down to maintain real-time processing speed. Then, DSSIM is performed on all areas of both images with a fixed window size.

[0082] Looking at Fig. 4, the result of the DSSIM-MAP generator measures dissimilarity by comparing the background image with the current image captured by the camera, and the brighter the area, the lower the similarity to the background. In other words, bright areas of similarity indicate that a new object has appeared that was not present in the background. Bright areas that differ from the background image in Fig. 3 may represent an object driving on a highway, i.e., a vehicle.

[0083] Figure 5 is a diagram showing the result of an ROI filter according to one embodiment.

[0084] The ROI filter can fill areas that are not the Region of Interest (ROI) with a meaningless solid color. Looking at Figure 5, the images in the left column are quantized by applying a threshold based on the DSSIM-MAP, filling the background area with white and the object area with black. The images in the right column are processed by utilizing the images in the left column to fill the object area with the original image's color. An appropriate image among the images in the right column can be transmitted to a server via a network.

[0085] Looking at Figure 5, the higher the threshold for dissimilarity, the narrower the area appears as the vehicle area, and the lower the threshold, the wider the vehicle area becomes. It can be seen that a low threshold also shows a tendency to segment areas that are not vehicles.

[0086] Figure 6 is a graph showing the transmission bit measurement results according to a threshold value according to one embodiment.

[0087] Figure 6 shows the bitrate generated for DSSIM values ​​according to the threshold used in the ROI filter, measured over time for about 7 seconds when a video to be transmitted to a server via a network is processed with an ROI filter and transmitted. The object detector receives an input image in which areas that do not need to be identified by the ROI filter are processed as solid colors. Since the areas replaced with solid colors are areas where no Image Feature is found, they can be ignored in the Regional Proposal stage.

[0088] While the bitrate of the original video is transmitted at 60kbps to 70kbps, the transmission rate of the filtered video is generally lower than that, showing that the transmission load reduction effect intended in the present invention can be achieved.

[0089] FIG. 7 is a drawing showing an example in which a predetermined color is filled into an area from which color has been removed according to one embodiment.

[0090] The object detector of the server can fill the empty areas of the received image with a specific color and perform object detection. Here, the empty areas refer to the areas removed by the intelligent CCTV and can represent the background. Figure 7(a) is an example where the empty areas are filled with black, and Figure 7(b) is an example where they are filled with green. By changing the non-ROI colors to highly saturated colors, the model can be processed so that it does not make region proposals during the inference process. Since black, white, and red are colors that can be used on vehicle surfaces or indicator lights and these colors can be confused with objects, highly saturated green (0,255,0) can be considered suitable as the color to fill the empty areas.

[0091] FIG. 8 is a diagram showing the inference time measurement results according to DSSIM and RPN threshold values ​​according to one embodiment.

[0092] To verify whether the image examined in Fig. 7 effectively removes areas unnecessary for object identification, the results of measuring the inference time while adjusting the RPN score threshold of the server-side model are as shown in the graph of Fig. 8.

[0093] When the RPN score threshold was not applied to the original Full HD image, it took approximately 28ms to infer one image. When the RPN score threshold was 0.1 or higher, it took approximately 25ms.

[0094] When the DSSIM Threshold of the ROI filter was 0.2 or higher, 26ms was taken in all cases, and the inference time tended to decrease as the DSSIM Threshold of the ROI filter increased. This demonstrates that when this technology is applied, the inference time of the server-side object detection model is shortened, enabling the processing of more images.

[0095] When examining the impact on object detection accuracy, when the DISSIM threshold is 0.3, background noise disappears when viewed visually, and the bitrate during transmission also remains at a consistently low level. Table 1 shows the Precision, Recall, F1, and mAP measured for each class detected by the object model.

[0096]

[0097] ClassPrecis-ionRecallF1mAPOriginalBus0.540.860.6776.32Car0.540.910.68Truck0.450.950.610.3+whiteBus0.620.810.7050.65Car0 .510.650.57Truck0.420.710.530.3+blackBus0.590.640.6240.14Car0.520.500.51Truck0.500.480.490.3+greenBus0.640.830.7266.20C ar0.590.910.72Truck0.650.550.600.3+white+erosionBus0.560.860.6869.45Car0.540.870.67Truck0.430.890.580.3+black+erosionBu s0.570.850.6870.98Car0.540.870.67Truck0.540.880.670.3+green+erosionBus0.550.880.6875.34Car0.560.900.69Truck0.590.900.71

[0098] In the existing case, the mAP was 40.14%, but when the non-ROI was filled with white or green, it increased to 50.65% and 66.20%, respectively. It can also be observed that the F1 score increased overall compared to black.

[0099] When erosion was performed, the percentages for white, black, and green were 69.45%, 70.98%, and 75.34%, respectively, which is higher than when erosion was not performed. These results indicate that securing the surrounding area helps with accurate detection.

[0100] It was confirmed that this technology can also operate effectively for object recognition, as it was possible to secure original-level accuracy when erosion was applied to the erased object regions of the DSSIM-MAP and the background regions were filled with green.

[0101] Meanwhile, the embodiments of the present invention described above can be written as a program executable on a computer and can be implemented on a general-purpose digital computer that operates the program using a computer-readable recording medium.

[0102] The above computer-readable recording media includes storage media such as magnetic storage media (e.g., ROM, floppy disk, hard disk, etc.) and optical reading media (e.g., CD-ROM, DVD, etc.).

[0103] Although embodiments of the present invention have been described above with reference to the attached drawings, those skilled in the art will understand that the present invention may be implemented in other specific forms without changing its technical concept or essential features. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive.

[0104] The present invention relates to an image preprocessing system and method for reducing transmission load in a network environment of an intelligent CCTV, and more specifically, to a technology for preprocessing images to transmit only the data necessary for object detection in order to reduce the transmission load in the intelligent CCTV and the processing load in the server.

Claims

1. In an image preprocessing system including an intelligent CCTV and a server, The above intelligent CCTV is, A camera that records video; A reference image generator that models a background image by accumulating the above images; A DSSIM-MAP generator that constructs a DSSIM-MAP by comparing the current image captured by the camera and the background image by region; An ROI filter that excludes the ROI and removes color based on the above DSSIM-MAP; and A communication unit that transmits a color-removed image to a server, excluding the above ROI; Includes, The above server is, A communication unit that receives video from the above intelligent CCTV; and A system comprising an object detector that detects an object by filling a predetermined color into an area from which the color has been removed in the received image.

2. In Paragraph 1, The above reference image generator is a system that generates a background image using Gaussian mixture modeling.

3. In Paragraph 1, The DSSIM-MAP for measuring the dissimilarity between the background image and the current image has Equation 1, The above mathematical formula 1 is (where x is the current image, y is the background image, is the average, is the standard deviation, and A system where is a constant.

4. In Paragraph 1, The above ROI filter sets a threshold value based on the above DSSIM-MAP, determines areas exceeding the threshold value as ROIs, and determines areas below the threshold value as non-ROIs, replaces non-ROIs with a solid color, and displays ROI areas in their original colors. In the above ROI filter, non-ROI is replaced with white or black having a higher compression ratio to reduce the size of the image transmitted to the server, and A system in which the object detector fills the area where the color has been removed with a color of higher saturation to further increase the object recognition rate.

5. In Paragraph 1, The above system is intended to track vehicle information using a CCTV network, and The aforementioned intelligent CCTV is installed on the road, and The above object is a system that includes a vehicle.

6. In Paragraph 1, The above intelligent CCTV is a system that uses Run-length encoding to reduce the capacity of the video transmitted to the server.

7. In Paragraph 1, A system in which the above reference image generator learns criteria for generating a background image from a plurality of images captured by the camera.

Images