Object detection device and object detection method

Abstract

This object detection device comprises: a detection unit that uses an object detection model to process an input image which includes a plurality of frames, detects a partial image specifying a thinning target, and outputs the results of object detection with respect to metadata from the input image, the metadata including at least the class, the reliability, and a bounding box of an object included in the input image; a correction unit that estimates a motion vector in the moving direction of the object, from the degree of coincidence between a current frame image and a past frame image in the input image, complements the object of the partial image serving as the thinning target by correcting the motion vector, and outputs the complementation results; and a synthesis unit that synthesizes the outputs the results of object detection and the complementation results, and outputs final detection results.

Description

Object detection device and object detection method

[0001] The disclosed technology relates to an object detection device and an object detection method.

[0002] Conventional technologies exist for object detection.

[0003] For example, there is the technology for object detection devices that estimate the class of an object (person, car, etc.), its bounding box (coordinate information of the rectangle surrounding the object), and its confidence level in an input image. In recent years, several object detection models using deep learning have been proposed. Examples of object detection models based on deep learning include YOLO (You Only Look Once), which infers the bounding box and object class together, and technologies using RetinaNet (Non-Patent Documents 1 and 2). In addition, R-CNN, which separates the detection of candidate object regions and the classification of classes, and its improved version, Faster R-CNN, have been proposed (Non-Patent Documents 3 and 4).

[0004] When performing object detection on high-resolution images or videos such as 4K or 8K, the input image size to the object detection model is limited to a maximum of 1536 x 1536 (YOLO v5), making it impossible to input the image at its original size. Therefore, a method has been proposed in which the high-resolution input image is divided to match the input image size of the object detection model, the results of object detection performed on the divided images are aggregated, and the result of object detection performed on the entire image is output. Several methods have been proposed depending on the image division method, but for example, one method is to input a group of images that are evenly divided to match the input size of the object detection model and a scaled-down image of the entire image to the object detection model (Non-Patent Document 5). In this method, the coordinate information of the obtained bounding boxes is scaled, the detection results of each divided and scaled-down image are combined, and the final result is output. Furthermore, a method has been proposed in which, on devices with limited computing resources such as edge terminals, the overall scaled-down and divided images to which object detection is applied are selected to reduce the amount of computation, and the areas to which object detection was not applied are filled in based on object detection results from past frames (Patent Document 1).

[0005] International Publication No. WO 2023 / 238248

[0006] J. Redmon et al., "You Only Look Once: Unified, Real-Time Object Detection," 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 779-788.T. Y. Lin et al., "Focal Loss for Dense Object Detection," 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2999-3007.R. Girshick et al., "Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation," 2014 IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 580-587.S. Ren et al., "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 6, pp. 1137-1149, 1 June 2017.H. Uzawa et al., “High-definition object detection technol-ogy based on AI inference scheme and its implementation,” IEICE Electronics Express, 2021, Volume 18, Issue 22, Pages 20210323.

[0007] However, conventional methods that use motion vectors to detect objects in past frames have a drawback: the coordinate correction results for objects may be inaccurate depending on errors in estimating the object's size or motion vector.

[0008] The disclosed technology was developed in view of the above points, and aims to provide an object detection device and object detection method that can improve the accuracy of coordinate correction in object detection.

[0009] The object detection device of this disclosure includes: a detection unit that processes an input image containing multiple frames using an object detection model, detects partial images that are to be thinned out, and outputs object detection results from the input image, including metadata such as the class, confidence level, and bounding box of the objects contained in the input image; a correction unit that estimates the motion vector of the object's movement direction from the degree of agreement between the current frame image and past frame images of the input image, corrects the motion vector for the objects in the thinned-out partial images to interpolate them, and outputs interpolation results; and a synthesis unit that synthesizes the output of the object detection results and the interpolation results to output a final detection result.

[0010] According to the disclosed technology, the accuracy of coordinate correction in object detection can be improved.

[0011] Figure 1 is a diagram illustrating the general processing of a conventional object detection device. Figure 2 is a diagram illustrating an example of a motion estimation method. Figure 3 is a block diagram showing the hardware configuration of the object detection device. Figure 4 is a block diagram showing the configuration of the object detection device of the first embodiment. Figure 5 is a flowchart showing the flow of object detection processing by the object detection device. Figure 6 is a flowchart showing the subroutine for the adjustment process of the first embodiment. Figure 7 is a diagram illustrating the adjustment of the motion vector. Figure 8 is a diagram illustrating the adjustment of the motion vector. Figure 9 is a block diagram showing the configuration of the object detection device of the second embodiment. Figure 10 is a flowchart showing the subroutine for the motion vector estimation process of the second embodiment. Figure 11 is a block diagram showing the configuration of the object detection device of the third embodiment.

[0012] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. In each drawing, identical or equivalent components and parts are given the same reference numerals. Furthermore, the dimensional ratios in the drawings are exaggerated for illustrative purposes and may differ from actual ratios.

[0013] First, an overview of this disclosure will be provided. The method exemplified in the conventional Patent Document 1, which serves as a premise, will be described. Figure 1 is a diagram illustrating the schematic of the processing of a conventional object detection device. The object detection device consists of three processing units: a detection unit, a correction unit, and a synthesis unit. In the detection unit, the input image is first reduced in size and divided. The division is performed evenly to match the input image size of the object detection model used. Next, thinning is performed to select the overall image, which is the reduced overall surface to which object detection is applied, and the partial images (divided images), which are the divided partial surfaces. At this time, thinning is performed so that object detection is applied to the overall image and divided images at regular frame intervals. As a result, the partial images are divided into partial images to be thinned and partial images not to be thinned. After that, object detection is performed on the overall image and partial images. As disclosed in the method of Patent Document 1, the detection result for the partial images not to be thinned is metadata obtained by performing object detection. The detection result for the partial images to be thinned is metadata obtained by correcting the position of the object using the estimated motion vector. The object detection results are used in the processing of the synthesis unit. The correction unit performs motion estimation from the input image and corrects the coordinates of the object detection results from past frames stored in the results record, reflecting this correction in the object detection results.

[0014] Figure 2 shows an example of a motion estimation method. (A1) p on the frame image at a certain time t-1 ｔ The pixel blocks on all four sides are to be searched. (A2) p on the frame image at time t ｓ The four surrounding areas are defined as the search area. The center coordinates of the target block and the search area on the input image are set to coincide. (A3) Within the search area, p ｔThe image is scanned by cutting out pixel blocks from all four sides and calculating the degree of agreement with the target block. As a result, the coordinate change to the pixel block in the search area with the highest degree of agreement is estimated as a motion vector. This motion vector is calculated for each pixel block in the image. Then, the object detection result at time t-1, which is stored in the result record, is corrected by estimating the coordinates at time t using the motion vector estimated from the target block closest to the center coordinates of each object. Finally, the compositing unit performs scaling of the object detection results from the detection unit, and combines the objects detected in the overall image (scaled down) and the divided partial images, as well as the object detection results output by the detection unit and the correction unit, to obtain the final object detection result. This object detection result is also stored as a result record in the correction unit and used for object detection in the frame at time t+1.

[0015] However, as mentioned in the above issues, there is a problem in that the coordinate correction results for objects may be inaccurate. In motion estimation, pixel blocks with a high degree of agreement with a search target block of a certain size are searched within the search range, but if the object is small or very large relative to the size of the block, it may not be possible to accurately estimate the motion vector. For example, the detection unit can detect a person with a width of less than 10 pixels, which is smaller than the pixel block size normally used. In this case, the center coordinates of the search target block may not be included within the bounding box of the object, and the motion vector estimated from the search target block whose center is outside the bounding box of the object will be selected, resulting in an inaccurate motion vector. Also, in the case of large objects such as trucks, if the area around the center coordinates of the object is a region with little change, such as a flatbed painted in a single color, the motion vector may not be detected.

[0016] Furthermore, if a pixel block with a higher degree of agreement than the actual motion vector direction appears within the search range, an incorrect motion vector may be estimated. If the object's coordinates are corrected with an incorrect motion vector, the center coordinates will shift from the correct object position. Once such a shift occurs, even if the motion vector is estimated correctly in subsequent frames, it becomes impossible to estimate the correct object coordinates.

[0017] In either of the above cases, it becomes impossible to estimate the precise coordinates of the object, which may reduce the accuracy of the final object detection result.

[0018] The method according to the embodiment of the present invention was made to solve the above-mentioned problems, and is an object detection device that outputs metadata as an object detection result, which includes at least the class, confidence level, and bounding box of the object contained in the input image from the input high-resolution input image. In the motion estimation process of the correction unit, the frame with the timestamp one timestamp before the current time is divided into search target blocks of a predetermined pixel size, and motion vectors are estimated. In the motion vector estimation, the area surrounding the search target block is used as the search area, and the area with the highest degree of agreement within the current frame is searched for, thereby estimating the motion vector representing the direction of movement of each search target block. In the correction unit of this technology, in addition to the conventional motion estimation, result storage, and coordinate correction processes, vector adjustment is performed. In vector adjustment, if the object detected in past frames is smaller or larger than a threshold, the sum of the motion vectors of each search target block that spans the bounding box (Bbox) of the object is calculated, or the motion vector with the largest absolute value among the motion vectors in Bbox is selected, thereby estimating the motion vector of the object according to the size of the object. Alternatively, the motion vectors from past frames are stored in the correction unit's result memory, and when compared with the motion vector estimated in the current frame, if the angle between the motion vectors from past frames is less than or equal to a threshold, the motion vector is calculated by taking the weighted sum of the two. Otherwise, the motion estimation in the current frame is considered to be incorrect, and the motion vector is corrected by using the motion vectors from past frames as they are.

[0019] Hereinafter, the configuration of this embodiment will be described. FIG. 3 is a block diagram showing the hardware configuration of the object detection device 100.

[0020] As shown in FIG. 3, the object detection device 100 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I / F) 17. Each component is connected to be communicable with each other via a bus 19.

[0021] The CPU 11 is a central processing unit that executes various programs and controls each part. That is, the CPU 11 reads a program from the ROM 12 or the storage 14 and executes the program using the RAM 13 as a work area. The CPU 11 performs control of the above components and various arithmetic processes according to the program stored in the ROM 12 or the storage 14. In this embodiment, an object detection program is stored in the ROM 12 or the storage 14.

[0022] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores a program or data as a work area. The storage 14 is composed of a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores various programs including an operating system and various data.

[0023] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to perform various inputs.

[0024] The display unit 16 is, for example, a liquid crystal display and displays various information. The display unit 16 may adopt a touch panel method and function as the input unit 15.

[0025] The communication interface 17 is an interface for communicating with other devices such as terminals. For this communication, for example, a standard for wired communication such as Ethernet (registered trademark) or FDDI, or a standard for wireless communication such as 4G, 5G, or Wi-Fi (registered trademark) is used.

[0026] Next, each functional configuration of the object detection device 100 will be described. FIG. 4 is a block diagram showing the configuration of the object detection device of the present embodiment. Each functional configuration is realized by the CPU 11 reading out the object detection program stored in the ROM 12 or the storage 14 and expanding and executing it in the RAM 13.

[0027] In the method of the present embodiment, in the correction unit 104, the motion vector is adjusted according to the size of the object, or the motion vector is monitored over a plurality of frames so that abnormal values are not used for coordinate correction. By this method, the accuracy of coordinate correction is improved, and the improvement of object detection accuracy in high-definition images and videos is realized. Hereinafter, embodiments will be described separately according to differences in the methods.

[0028] [First Embodiment] As shown in FIG. 4, the object detection device 100 includes a detection unit 102, a correction unit 104, and a synthesis unit 106. The basic configuration of the object detection device 100 of the present embodiment includes a processing unit similar to the conventional object detection device shown in FIG. 1 described above, but the processing mode is different. In the first embodiment, it is different from the prior art in that the correction unit 104 performs vector adjustment to adjust the vector obtained by motion estimation according to the size of the object. An input image including a plurality of frames is input to the object detection device 100.

[0029] The detection unit 102 processes the input image using an object detection model and detects partial images that have been identified as targets for thinning. In the detection of partial images, the overall reduced image and the partial images of targets for thinning and non-thinning are detected, as illustrated in Figure 1. The detection unit 102 also outputs object detection results from the input image, which include metadata such as the class, confidence level, and bounding box of the objects contained in the input image. The object detection results are the object detection results for both the overall image and the partial images of the entire surface, and are detected using, for example, the method described in Patent Document 1.

[0030] The correction unit 104 estimates the motion vector of an object's movement direction from the degree of agreement between the current frame image and past frame images among the input images. The current frame image is the frame image currently being processed. The correction unit 104 interpolates the motion vectors of objects in the partial images that have been downsampled and outputs the interpolation results. Specifically, in the motion vector correction, processing is performed for a first object smaller than a predetermined size and a second object larger than a predetermined size. The motion vector of the first object smaller than a predetermined size is determined from the average of the motion vectors obtained from each of the search target blocks that span the object's bounding box. For the second object larger than a predetermined size, the motion vector with the largest absolute value is selected from among the motion vectors obtained from multiple search target blocks within the bounding box. In this way, the motion vector is corrected according to the size of the object.

[0031] The synthesis unit 106 synthesizes the object detection result and the size-based interpolation result to output the final detection result.

[0032] Next, the operation of the object detection device 100 of the first embodiment will be described. Figure 5 is a flowchart showing the flow of object detection processing by the object detection device 100. Object detection processing is performed when the CPU 11 reads the object detection program from the ROM 12 or storage 14, loads it into the RAM 13, and executes it. The object detection device 100 receives an input image as input and performs the following processing.

[0033] In step S100, as the detection unit 102, the CPU 11 performs processing using an object detection model on the input image and detects a partial image that identifies the thinning target.

[0034] In step S102, as the detection unit 102, the CPU 11 outputs an object detection result of metadata including at least the class, reliability, and bounding box of the object included in the input image from the input image.

[0035] In step S104, as the correction unit 104, the CPU 11 estimates a motion vector in the moving direction of the object from the degree of coincidence between the current frame image and the past frame image in the input image.

[0036] In step S106, as the correction unit 104, the CPU 11 performs correction of the motion vector for the object in the partial image that is the thinning target, completes it, and outputs the completion result. The adjustment process of this step will be described later as a subroutine.

[0037] In step S108, as the synthesis unit 106, the CPU 11 synthesizes the object detection result and the completion result according to the size and outputs the final detection result.

[0038] Next, the subroutine of the adjustment process in step S106 will be described. FIG. 6 is a flowchart showing the subroutine of the adjustment process of the first embodiment.

[0039] In the flowchart, first, processing is started for each object in the past frame image of the result record. Let the size of the search target block in motion estimation be p ｔ , and the size of the search range be p ｓ . The method of setting the threshold for determining the adjustment method is arbitrary, but the number of vertical or horizontal pixels of the bounding box of the object (hereinafter referred to as Bbox) is (i) less than p ｔ , (ii) greater than or equal to p ｔ and less than p ｓ , and (iii) three patterns when it is greater than or equal to p ｓ are assumed.

[0040] In step S120, the CPU 11 determines that the length or width of the object is p ｔ Determine if it is less than a pixel. ｔ If it is less than a pixel, proceed to step S122, p ｔ If it is not less than a pixel, proceed to step S124. That is, if the predetermined size is p ｔ The process that proceeds to step S122 involves processing the first object, which is small in size, as a pixel, while the process that proceeds to step S124 involves processing the second object, which is larger in size.

[0041] In step S122, the CPU 11 calculates an integrated motion vector from the motion vectors of all search target blocks that span Bbox.

[0042] In case (i), pixels not included in the object's Bbox are included in the searched block, which may cause an error between the object's original direction of movement and the motion vector to the pixel block with the highest degree of agreement within the search range. Therefore, in order to estimate the motion vector more accurately based on the motion estimation results in multiple searched blocks, the average of the motion vectors estimated in each searched block that the object's Bbox spans is taken. Figure 7 is a diagram to explain the adjustment of the motion vector. For example, as shown in Figure 7, Bbox is in two searched blocks B １ and B ２ If it spans across multiple blocks, the motion vector V of the object is the motion vector V estimated in each searched block. １ and V ２ The calculation is performed as follows: ... (1)

[0043] This example shows the case where the search spans two target blocks, but if it spans n target blocks, the motion vector of the object is calculated by averaging the motion vectors estimated across all target blocks. In this case, a weighted average may be calculated instead of a simple average. In that case, for example, the weighting is done by the proportion of the area of ​​the overlapping portion between Bbox and the target block that occupies the area of ​​Bbox. That is, target block B ｉThe ratio of the area of ​​the overlapping part of Bbox to the area of ​​Bbox is λ. ｉ It can be calculated using the following formula. ... (2)

[0044] In step S124, the CPU 11 determines that the length or width of the object is p ｓ Determine if it is greater than or equal to a pixel. ｓ If it is not more than a pixel, proceed to step S126, p ｓ If the number is greater than or equal to 10 pixels, proceed to step S128.

[0045] In step S126, the CPU 11 uses the obtained motion vector as is (in case (ii)).

[0046] In step S128, the CPU 11 selects multiple target blocks within Bbox and calculates motion vectors.

[0047] In case (iii), the object is large, and the pixels of the search target block containing the center of Bbox are almost the same color and lack features, making it possible that the motion vector cannot be correctly estimated. Therefore, as shown in Figure 8, multiple search target blocks with their center within the object's Bbox are selected, and each search target block (B １ ~B ４ The motion vector (V) estimated from ) １ ~V ４ The motion vector (V) of the object is calculated from the above. The method of selecting the blocks to be searched is arbitrary, but you may select the blocks to be searched at the four corners of Bbox or the blocks to be searched that contain the midpoints of each side, or you may select the blocks to be searched inside Bbox, such as the blocks to be searched that contain the midpoints of the perpendiculars drawn from the center of Bbox to each side. The motion vector may be calculated by calculating the average of the motion vectors of each block to be searched, as in (i). Alternatively, it may be calculated by selecting one from among several motion vectors. For example, one method is to adopt the motion vector with the largest absolute value as the motion vector of the object.

[0048] Furthermore, in addition to calculating the motion vector as described above, the motion vector may be adjusted by a constant multiplier according to the size of the object. This method of adjusting the motion vector by a constant multiplier is particularly effective for large, fast-moving objects such as trucks and cars, as they may be moving beyond the target block being searched.

[0049] Then, once the adjustment of the motion vectors of all objects is complete, in step S130, the CPU 11 corrects the position of the objects using the adjusted motion vectors.

[0050] In this way, vector adjustment makes it possible to estimate motion vectors more accurately, and even when decimation is performed, the object detection results can be supplemented using object detection results from past frames, thereby improving their accuracy.

[0051] As described above, the object detection device 100 of this embodiment can improve the accuracy of coordinate correction in object detection.

[0052] [Second Embodiment] As shown in Figure 9, the object detection device 200 is configured to include a detection unit 102, a correction unit 204, and a synthesis unit 106. In the second embodiment, the correction unit 204 stores the motion vector estimated by motion estimation and uses the motion vector stored for past frame images to correct the motion vector in the current frame image. Thus, it differs in that, in addition to the object detection result output by the synthesis unit 106, the result obtained in motion estimation is stored as a result record and used to estimate the motion vector of the next frame and beyond. In motion estimation, not only is the degree of agreement between the current frame and the frame with the previous timestamp measured, but the motion vector in the current frame is also corrected using the motion estimation result for past frames stored in the result memory.Hereafter, parts that are the same as in the first embodiment are denoted by the same reference numerals and their explanation is omitted, and only the different processing modes will be described.

[0053] In the second embodiment, the process in step S104 is replaced with the motion vector estimation subroutine shown in Figure 10, and the adjustment processes in steps S120 to S128 are omitted. Figure 10 is a flowchart of the motion vector estimation subroutine in the second embodiment. In the second embodiment, the motion vector in the current frame image is corrected using the motion vector in past frame images.

[0054] In step S210, the CPU 11 first estimates the motion in a target block from the current frame and the frame with the previous timestamp, using a method similar to that of conventional methods, and obtains a motion vector.

[0055] Next, in step S212, the CPU 11 reads the motion vector of the same search target block stored in the result record. Here, the motion vector estimated in motion estimation is V ｃｕｒｒｅｎｔ The motion vector read from the result record is V ｐａｓｔ Let's assume that. Next, we calculate the angle θ formed by each motion vector using the following formula. ... (3)

[0056] Next, in step S214, the CPU 11 determines whether the angle between the two motion vectors is less than or equal to a threshold. If it is less than or equal to the threshold, the process proceeds to step S216; otherwise, the process proceeds to step S218.

[0057] In step S216, the CPU 11 calculates the motion vector of the block being searched for from both motion vectors. In step S218, the past motion vector is adopted as the motion vector of the block being searched for.

[0058] The angle formed is the threshold θ ｔｈｒｅｓｈ If it is larger, V ｃｕｒｒｅｎｔ Assuming that is an estimation error, V ｐａｓｔ The motion vector V of the block to be searched is adopted (step S218). On the other hand, θ is the threshold θ ｔｈｒｅｓｈ If the following conditions are met, V ｃｕｒｒｅｎｔ and V ｐａｓｔThe motion vector V is calculated by combining these (step S216). For example, one calculation method is to calculate the weighted average of the vectors. The following equation is given by λ ｃｕｒｒｅｎｔ , λ ｐａｓｔ These are weights that can be arbitrarily set so that their sum equals 1. ... (4)

[0059] Subsequently, the motion vector is saved in the result record (step S220), and the subsequent coordinate correction process is performed. In this example, the motion vector V estimated in the current frame is used. ｃｕｒｒｅｎｔ And the motion vector V in the frame of the previous timestamp. ｐａｓｔ The weighted average is calculated, but this is not the only method for calculating motion vectors. Another calculation method is V ｃｕｒｒｅｎｔ and V ｐａｓｔ One possible approach is to take the maximum value of [the specified value]. Alternatively, one could store the motion vectors calculated using conventional motion estimation in a results memory for a certain period and then calculate their moving average.

[0060] As described above, when the correction unit 204 corrects the motion vector in the current frame using the motion vector in past frame images, it compares the angles of the motion vectors. If the angle between the motion vector in past frame images and the motion vector estimated in the current frame is less than or equal to a certain value, the correction unit 204 determines the motion vector in the current frame image by taking a weighted sum of the two. If the angle is greater than or equal to a certain value, the correction unit 204 considers the motion vector estimated in the current frame to be an outlier. Then, it corrects the motion vector by adopting the motion vector in past frames as the motion vector in the current frame. In this way, by correcting the motion vector using the results of motion estimation in past frames, the impact of errors in motion vector estimation is reduced, the direction of movement of objects can be estimated more accurately, and the accuracy of object detection is improved.

[0061] [Third Embodiment] Figure 11 shows a configuration diagram that realizes the third embodiment. This embodiment is equipped with both the internal devices of the estimation unit shown in the first and second embodiments described above, and performs the processing shown in each embodiment together. By performing both motion estimation results in past frames and correction of motion vectors based on the size of the object, it becomes possible to predict the direction of movement adaptively for each object while eliminating errors in motion vector estimation, thereby maximizing object detection accuracy.

[0062] In other words, the correction unit 304 of the object detection device 300 stores the motion vectors estimated by motion estimation, and uses the motion vectors stored for past frame images to correct the motion vectors in the current frame image. Furthermore, for the corrected motion vectors, the motion vector of a first object smaller than a predetermined size is determined from the average of the motion vectors obtained from each of the search target blocks that the object's bounding box spans, and the motion vector of a second object larger than a predetermined size is further corrected by selecting the motion vector with the largest absolute value among the motion vectors obtained from multiple search target blocks within the bounding box.

[0063] In addition, the object detection process that the CPU reads and executes in each of the above embodiments may be executed by various processors other than the CPU. Examples of such processors include dedicated electrical circuits, which are processors with circuit configurations specifically designed to execute particular processes, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices) whose circuit configuration can be changed after manufacturing, GPUs (Graphics Processing Units), and ASICs (Application Specific Integrated Circuits). Furthermore, the object detection process may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (for example, multiple FPGAs, and a combination of a CPU and an FPGA). Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit composed of circuit elements such as semiconductor devices.

[0064] Furthermore, although the above embodiments describe an embodiment in which the object detection program is pre-stored (installed) in the storage 14, the invention is not limited to this. The program may be provided in a form stored on a non-transitor storage medium such as a CD-ROM (Compact Disk Read Only Memory), DVD-ROM (Digital Versatile Disk Read Only Memory), or USB (Universal Serial Bus) memory. Alternatively, the program may be provided in a form downloaded from an external device via a network.

[0065] The following additional information is disclosed regarding the embodiments described above.

[0066] (Note 1) An object detection device comprising: a memory; and at least one processor connected to the memory, wherein the processor processes an input image containing multiple frames using an object detection model, detects partial images that are to be downsampled, outputs an object detection result from the input image including at least the class, confidence level, and bounding box of the object contained in the input image; estimates the motion vector of the object's movement direction from the degree of agreement between the current frame image and past frame images of the input image; complements the motion vector for the object in the partial image that has been downsampled by correcting it, outputs the complementation result; and outputs a final detection result by combining the output of the object detection result and the complementation result.

[0067] (Appendix 2) A non-temporary storage medium that stores a program executable by a computer to perform object detection processing, wherein it processes an input image containing multiple frames using an object detection model, detects partial images that are to be thinned out, outputs object detection results from the input image including metadata such as the class, confidence level, and bounding box of the objects contained in the input image, estimates the motion vector of the object's direction of movement from the degree of agreement between the current frame image and past frame images of the input image, compensates for the objects in the partial images that were to be thinned out by correcting the motion vector, outputs the compensation results, and outputs a final detection result by combining the output of the object detection results and the compensation results.

[0068] 100, 200, 300 Object detection device 102 Detection unit 104, 204, 304 Correction unit 106 Synthesis unit

Claims

1. An object detection device comprising: a detection unit that processes an input image containing multiple frames using an object detection model, detects partial images that are to be downsampled, and outputs object detection results from the input image, including metadata that at least includes the class, confidence level, and bounding box of the object contained in the input image; a correction unit that estimates the motion vector of the object's movement direction from the degree of agreement between the current frame image and past frame images of the input image, corrects the motion vector for the object in the partial image that was to be downsampled, and outputs the completion result; and a synthesis unit that synthesizes the output of the object detection result and the completion result to output a final detection result.

2. The object detection device according to claim 1, wherein the correction unit corrects the motion vector by determining the motion vector of a first object smaller than a predetermined size from the average of motion vectors obtained from each of the search target blocks spanning the bounding box of the object, and by selecting the motion vector with the largest absolute value among the motion vectors obtained from a plurality of search target blocks within the bounding box for a second object larger than a predetermined size.

3. The object detection device according to claim 1, wherein the correction unit stores motion vectors estimated by motion estimation and corrects the motion vectors in the current frame image using the motion vectors stored for past frame images.

4. An object detection method performed by a computer, comprising: processing an input image containing multiple frames using an object detection model to detect partial images that are to be downsampled; outputting object detection results from the input image, including metadata such as the class, confidence level, and bounding box of the objects contained in the input image; estimating the motion vector of the object's movement direction from the degree of agreement between the current frame image and past frame images of the input image; correcting the motion vector for the objects in the downsampled partial images to interpolate them and outputting interpolation results; and synthesizing the output of the object detection results and the interpolation results to output a final detection result.

Images