Apparatus and method for detecting object from image

Abstract

The present disclosure relates to an apparatus and method for detecting an object from an image. The method for detecting an object from an image, according to an embodiment, may comprise an operation of generating a plurality of bounding boxes corresponding to the object. The method may comprise an operation of generating, on a camera coordinate system of a camera that has been used to capture the image, a ray connecting an origin and a first bounding box from among the plurality of bounding boxes. The method may comprise an operation of generating, on the basis of the ray, a frustum comprising the first bounding box therein. The method may comprise an operation of removing, on the basis of the frustum, at least one second bounding box other than the first bounding box from among the bounding boxes.

Description

Device and method for detecting objects from an image

[0001] The present disclosure relates to an apparatus and method for detecting an object from an image.

[0002] Object detection is a technology that detects objects of a specific class (e.g., people, vehicles, or animals) from an image and generates (or outputs) bounding boxes that provide information about the location of the detected objects.

[0003] For example, in an autonomous driving system, object detection can be used to recognize various objects located around the vehicle (e.g., pedestrians or other vehicles).

[0004] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0005] In one embodiment, in camera-based 3D object detection, the issue of duplicate detection that may occur along the depth dimension can be resolved.

[0006] The technical tasks intended to be accomplished in this document are not limited to those mentioned above, and other technical tasks not mentioned will be clearly understood by those skilled in the art to which this document belongs from the description below.

[0007] An apparatus for detecting an object from an image according to one embodiment may include at least one processor and a memory for storing instructions. When the instructions are executed individually or collectively by the at least one processor, the apparatus may perform a plurality of operations. The plurality of operations may generate a plurality of bounding boxes corresponding to the object. The plurality of operations may generate a ray connecting an origin and a first bounding box among the plurality of bounding boxes in the camera coordinate system of a camera used to capture the image. The plurality of operations may include an operation to remove at least one second bounding box among the bounding boxes, excluding the first bounding box, based on the frustum.

[0008] The first bounding box may have the highest confidence score among the plurality of bounding boxes. The confidence score of each of the plurality of bounding boxes may be a probabilistic score indicating the probability that each of the plurality of bounding boxes corresponds to the object.

[0009] The above frustum may have the above ray as a central axis.

[0010] The operation of removing at least one second bounding box may include removing at least one second bounding box located within the frustum among the bounding boxes.

[0011] The above ray can pass through the origin and the center of the first bounding box.

[0012] The above camera may be mounted on a vehicle.

[0013] The above object may be a pedestrian.

[0014] A method for detecting an object from an image according to one embodiment may include the operation of generating a plurality of bounding boxes corresponding to the object. The method may include the operation of generating a ray connecting an origin and a first bounding box among the plurality of bounding boxes on the camera coordinate system of a camera used to capture the image. The method may include the operation of generating a frustum containing the first bounding box based on the ray. The method may include the operation of removing at least one second bounding box among the bounding boxes, excluding the first bounding box, based on the frustum.

[0015] The first bounding box may have the highest confidence score among the plurality of bounding boxes. The confidence score of each of the plurality of bounding boxes may be a probabilistic score indicating the probability that each of the plurality of bounding boxes corresponds to the object.

[0016] The above frustum may have the above ray as a central axis.

[0017] The operation of removing at least one second bounding box may include removing at least one second bounding box located within the frustum among the bounding boxes.

[0018] The above ray can pass through the origin and the center of the first bounding box.

[0019] The above camera may be mounted on a vehicle.

[0020] The above object may be a pedestrian.

[0021] According to one embodiment, a computer-readable recording medium storing one or more computer programs may include instructions for performing the method in a processor.

[0022] FIG. 1 is a diagram illustrating input data and output data of an electronic device according to one embodiment.

[0023] Figure 2 is a diagram illustrating NMS.

[0024] FIG. 3 is a drawing for explaining a frustum for an NMS according to one embodiment.

[0025] FIG. 4 is a flowchart for explaining an NMS according to one embodiment.

[0026] FIG. 5 is a schematic block diagram of an electronic device according to one embodiment.

[0027] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Accordingly, actual implementations are not limited to the specific embodiments disclosed, and the scope of this specification includes modifications, equivalents, or substitutions included in the technical concept described by the embodiments.

[0028] Terms such as "first" or "second" may be used to describe various components, but these terms should be interpreted solely for the purpose of distinguishing one component from another. For example, the first component may be named the second component, and similarly, the second component may be named the first component.

[0029] When it is stated that a component is "connected" to another component, it should be understood that it may be directly connected to or coupled with that other component, or that there may be other components in between.

[0030] Singular expressions include plural expressions unless the context clearly indicates otherwise. In this document, phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B or C,” “at least one of A, B and C,” and “at least one of A, B, or C” may each include any one of the items listed together with the corresponding phrase, or all possible combinations thereof. In this specification, terms such as “comprising” or “having” are intended to designate the existence of the described feature, number, step, action, component, part, or combination thereof, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0031] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this specification.

[0032] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. It should be noted that the embodiments of the present disclosure may be referenced, borrowed, or combined with one another. In the description with reference to the accompanying drawings, identical components are given the same reference numeral regardless of the drawing number, and redundant descriptions thereof will be omitted.

[0033]

[0034] FIG. 1 is a diagram illustrating input data and output data of an electronic device according to one embodiment.

[0035] Referring to FIG. 1, according to one embodiment, an electronic device can detect objects from image data. The electronic device can detect objects of various classes (e.g., people, vehicles, or animals) within the image data and generate (or output) bounding boxes that provide location information of the objects in three-dimensional space.

[0036] The electronic device can be mounted on a vehicle (e.g., an autonomous vehicle). To provide information about the environment surrounding the vehicle, the electronic device can detect objects from image data acquired through a camera sensor mounted on the vehicle and generate bounding boxes corresponding to the objects.

[0037] According to one embodiment, the electronic device may apply non-maximum suppression (NMS) to resolve the issue of duplicate detection in the depth dimension that may occur in 3D object detection based on image data.

[0038]

[0039] Figure 2 is a diagram illustrating NMS.

[0040] Referring to FIG. 2, according to one embodiment, an electronic device (e.g., the electronic device (100) of FIG. 1) may generate a bounding box to detect an object (e.g., a pedestrian or an animal) from an image. The bounding box may provide location information of the object within the image. Although FIG. 2 illustrates an NMS in a two-dimensional space, an NMS in a three-dimensional space may also be substantially the same.

[0041] At least one bounding box may be generated for a specific object within an image. For example, in 3D object detection, if depth estimation based on the image is incomplete, multiple bounding boxes may be generated for the same object along the depth dimension.

[0042] When multiple bounding boxes are generated for a specific object within an image, the electronic device may generate a confidence score for each of the multiple bounding boxes. The confidence score may be a probabilistic score representing the probability that each of the multiple bounding boxes corresponds to a specific object (or the probability that each of the multiple bounding boxes contains a specific object). For example, if three bounding boxes are generated for a pedestrian within an image, the pedestrian may be most likely to be contained within the bounding box with the highest confidence score.

[0043] In object detection, duplicate bounding boxes for the same object can degrade performance, so non-maximum suppression (NMS) may be necessary to remove duplicate bounding boxes.

[0044]

[0045] FIG. 3 is a drawing for explaining a frustum for an NMS according to one embodiment.

[0046] Referring to FIG. 3, according to one embodiment, an electronic device (e.g., the electronic device (100) of FIG. 1) may generate a frustum to remove duplicate bounding boxes for the same object within an image. It should be noted that the number of bounding boxes and the size of the frustum shown in FIG. 3 are examples for illustrating the present disclosure.

[0047] When multiple bounding boxes (32–38) are generated for a specific object (e.g., a pedestrian) within an image, the electronic device may generate a frustum (44) to remove at least one of the multiple bounding boxes (32–38).

[0048] The electronic device may generate a ray (42) (e.g., a virtual line) on the camera coordinate system (e.g., a 3D coordinate system) of the camera used to acquire the image in order to generate a frustum. The electronic device may generate a ray (42) passing through the origin (20) of the camera coordinate system and any one of a plurality of bounding boxes (32 to 38). For example, the electronic device may generate a ray (42) connecting the center of the bounding box (e.g., bounding box (34)) having the highest confidence score among the plurality of bounding boxes (32 to 38) and the origin (20).

[0049] The electronic device can generate a frustum (44) having a beam (42) as a central axis. The size of the frustum can be determined based on an ablation study.

[0050]

[0051] FIG. 4 is a flowchart for explaining an NMS according to one embodiment.

[0052] Referring to FIG. 4, according to one embodiment, an electronic device (e.g., the electronic device (100) of FIG. 1) can resolve redundant bounding box issues through a frustum-based NMS.

[0053] In operation 410, the electronic device can generate a frustum (e.g., frustum (44) of FIG. 3) containing multiple bounding boxes (e.g., bounding boxes (32–38) of FIG. 3) for the same object (e.g., pedestrian).

[0054] In operation 420, the electronic device may remove at least one of the bounding boxes within the frustum (e.g., bounding boxes (32–36) of FIG. 3). For example, the electronic device may remove the remaining bounding boxes (e.g., bounding boxes (32, 36) of FIG. 3), excluding the bounding box with the highest reliability score among the bounding boxes within the frustum (e.g., bounding box (34) of FIG. 3).

[0055] According to one embodiment, an electronic device can resolve the issue of duplicate detection in depth dimensions that may arise in image-based 3D object detection through frustum-based NMS. For example, when detecting small objects such as pedestrians included in an image, multiple duplicate bounding boxes may be generated due to noise and / or inaccuracy in depth estimation. The electronic device can detect objects within an image with high accuracy by removing duplicate bounding boxes through frustum-based NMS.

[0056]

[0057] FIG. 5 is a schematic block diagram of an electronic device according to one embodiment.

[0058] Referring to FIG. 5, according to one embodiment, the electronic device (100) may include at least one processor (520) and a memory (540).

[0059] The memory (540) may store instructions (or programs) executable by at least one processor (520). For example, the instructions may include instructions for executing the operation of at least one processor (520) and / or the operation of each configuration of at least one processor (520).

[0060] The memory (540) may include one or more computer-readable storage media. The memory (540) may include non-volatile storage devices (e.g., magnetic hard disc, optical disc, floppy disc, flash memory, EPROM (electrically programmable memories), EEPROM (electrically erasable and programmable)).

[0061] The memory (540) may be a non-transitory medium. The term "non-transitory" may indicate that the storage medium is not implemented by a carrier wave or a propagated signal. However, the term "non-transitory" should not be interpreted as meaning that the memory (540) is immobile.

[0062] At least one processor (520) can process data stored in memory (540). At least one processor (520) can execute computer-readable code (e.g., software) stored in memory (540) and instructions triggered by at least one processor (520).

[0063] At least one processor (520) may be a data processing device implemented in hardware having a circuit having a physical structure for executing desired operations. For example, the desired operations may include code or instructions included in a program.

[0064] For example, a data processing device implemented in hardware may include a microprocessor, a central processing unit, a processor core, a multi-core processor, a multiprocessor, an Application-Specific Integrated Circuit (ASIC), and a Field Programmable Gate Array (FPGA).

[0065] At least one processor (520) may include a main processor (e.g., a central processing unit or an application processor) and an auxiliary processor (e.g., a communication processor, a neural processing unit (NPU), and / or a graphic processing unit (GPU)).

[0066] At least one processor (520) can enable the electronic device (100) to perform at least one operation by individually or collectively executing code, instructions, and / or applications stored in memory (540).

[0067]

[0068] The embodiments described above may be implemented as hardware components, software components, and / or combinations of hardware and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0069] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be stored on any type of machine, component, physical device, virtual equipment, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on computer-readable recording media.

[0070] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may store program instructions, data files, data structures, etc., either individually or in combination, and the program instructions recorded on the medium may be those specifically designed and configured for the embodiment or those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0071] The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0072] Although the embodiments have been described above with reference to the limited drawings, those skilled in the art can apply various technical modifications and variations based thereon. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0073] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

1. In a device for detecting an object from an image, At least one processor; and memory that stores instructions Includes, When the above instructions are executed individually or collectively by the at least one processor, the device is made to perform a plurality of operations, and The above plurality of operations are, The operation of generating a plurality of bounding boxes corresponding to the above object; An operation to generate a ray connecting the origin and the first bounding box among the plurality of bounding boxes in the camera coordinate system of the camera used to capture the above image; Based on the above light rays, the operation of generating a frustum containing the first bounding box inside; and Based on the above frustum, an operation to remove at least one second bounding box among the bounding boxes, excluding the first bounding box. A device including 2. In Paragraph 1, The above first bounding box is, Having the highest confidence score among the above plurality of bounding boxes, The reliability score of each of the above plurality of bounding boxes is, A device in which each of the plurality of bounding boxes is a probabilistic score representing the probability that it corresponds to the object.

3. In Paragraph 1, The above frustum is, A device having the above-mentioned ray as a central axis.

4. In Paragraph 1, The operation of removing at least one second bounding box is, The operation of removing at least one second bounding box located within the frustum among the bounding boxes. A device including 5. In Paragraph 1, The above ray is, A device passing through the origin and the center of the first bounding box.

6. In Paragraph 1, The above camera is a device mounted on a vehicle.

7. In Paragraph 1, The above object is, Pedestrian, device.

8. A method for detecting objects from an image, The operation of generating a plurality of bounding boxes corresponding to the above object; An operation to generate a ray connecting the origin and the first bounding box among the plurality of bounding boxes in the camera coordinate system of the camera used to capture the above image; Based on the above light rays, the operation of generating a frustum containing the first bounding box inside; and Based on the above frustum, an operation to remove at least one second bounding box among the bounding boxes, excluding the first bounding box. A method including 9. In Paragraph 8, The above first bounding box is, Having the highest reliability score among the above plurality of bounding boxes, The reliability score of each of the above plurality of bounding boxes is, A method in which each of the above plurality of bounding boxes is a probabilistic score representing the possibility that it corresponds to the object.

10. In Paragraph 8, The above frustum is, A method having the above-mentioned ray as a central axis.

11. In Paragraph 8, The operation of removing at least one second bounding box is, The operation of removing at least one second bounding box located within the frustum among the bounding boxes. A device including 12. In Paragraph 8, The above ray is, A method passing through the origin and the center of the first bounding box.

13. In Paragraph 8, The above camera is mounted on a vehicle, a method.

14. In Paragraph 8, The above object is, Pedestrian, device.

15. A computer program stored on a computer-readable recording medium in combination with hardware to execute the method of any one of claims 8 through 14.

Images