Apparatus and method for calibrating camera sensor and lidar sensor

Abstract

Disclosed are an apparatus and method for calibrating a camera sensor and a LiDAR sensor. The method for calibrating a camera sensor and a LiDAR sensor may comprise an operation of, while a vehicle located on a turn table is rotated by a turn table, obtaining, through the camera sensor mounted on the vehicle, image data including a marker pattern image of a calibration board mounted on a robot arm located around the vehicle. The method may comprise an operation of moving the robot arm to a waypoint on a set trajectory while the data is obtained. The method may comprise an operation of obtaining, through the LiDAR sensor mounted on the vehicle, point cloud data including points corresponding to the calibration board while the vehicle is rotated by the turn table. The method may comprise an operation of calibrating the camera sensor and the LiDAR sensor on the basis of the image data and the point cloud data.

Description

Device and method for calibrating camera sensors and LiDAR sensors

[0001] The present disclosure relates to an apparatus and method for calibrating a camera sensor and a lidar sensor.

[0002] LiDAR sensors and camera sensors may be used for the operation of autonomous vehicles. When LiDAR sensors and camera sensors are mounted at different locations on the vehicle, a calibration process may be required to align image data with point cloud data.

[0003] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. None of the foregoing information shall be applied as prior art related to the present disclosure.

[0004] One embodiment may provide an automated calibration system using a turntable and a robot arm.

[0005] One embodiment can provide a method for calibrating camera sensors and LiDAR sensors with high precision.

[0006] The technical tasks intended to be accomplished in this document are not limited to those mentioned above, and other technical tasks not mentioned will be clearly understood by those skilled in the art to which this document belongs from the description below.

[0007] An apparatus for calibrating a camera sensor and a LiDAR sensor according to one embodiment may include at least one processor and a memory for storing instructions. When the instructions are executed individually or collectively by the at least one processor, the apparatus may be able to perform a plurality of operations. The plurality of operations may include an operation of acquiring image data including a marker pattern image of a calibration board mounted on a robot arm located around the vehicle, through the camera sensor mounted on the vehicle while the vehicle located on the vehicle turntable is rotated by the turntable. The plurality of operations may include an operation of acquiring point cloud data including points corresponding to the calibration board, through the LiDAR sensor mounted on the vehicle while the vehicle is rotated by the turntable. The plurality of operations may include an operation of calibrating the camera sensor and the LiDAR sensor based on the image data and the point cloud data.

[0008] The above image data may include a first image frame captured when the turntable is rotated by a first angle and a second image frame captured when the turntable is rotated by a second angle.

[0009] The above point cloud data may include a first point cloud frame corresponding to the first image frame and a second point cloud frame corresponding to the second image frame.

[0010] The first image frame may include a first marker pattern image of the calibration board when the robot arm is positioned at a first point on the set trajectory. The second image frame may include a second marker pattern image of the calibration board when the robot arm is positioned at a second point on the set trajectory.

[0011] The calibration operation described above may include an operation to determine a first normal vector of the calibration board based on the image data. The calibration operation may include an operation to determine a second normal vector of the calibration board based on the point cloud data. The calibration operation may include an operation to calibrate the camera sensor and the LiDAR sensor based on the first normal vector and the second normal vector.

[0012] The above calibration operation may include an operation to simultaneously calibrate a plurality of camera sensors and a plurality of LiDAR sensors.

[0013] A method for calibrating a camera sensor and a lidar sensor may include the operation of acquiring image data including a marker pattern image of a calibration board mounted on a robot arm located around the vehicle, through the camera sensor mounted on the vehicle while the vehicle located on the turntable is rotated by the turntable. The method may include the operation of acquiring point cloud data including points corresponding to the calibration board, through the lidar sensor mounted on the vehicle while the vehicle is rotated by the turntable. The method may include the operation of calibrating the camera sensor and the lidar sensor based on the image data and the point cloud data.

[0014] The above image data may include a first image frame captured when the turntable is rotated by a first angle and a second image frame captured when the turntable is rotated by a second angle.

[0015] The above point cloud data may include a first point cloud frame corresponding to the first image frame and a second point cloud frame corresponding to the second image frame.

[0016] The first image frame may include a first marker pattern image of the calibration board when the robot arm is positioned at a first point on the set trajectory. The second image frame may include a second marker pattern image of the calibration board when the robot arm is positioned at a second point on the set trajectory.

[0017] The calibration operation described above may include an operation to determine a first normal vector of the calibration board based on the image data. The calibration operation may include an operation to determine a second normal vector of the calibration board based on the point cloud data. The calibration operation may include an operation to calibrate the camera sensor and the LiDAR sensor based on the first normal vector and the second normal vector.

[0018] The above calibration operation may include an operation to simultaneously calibrate a plurality of camera sensors and a plurality of LiDAR sensors.

[0019] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0020] FIG. 1 is a diagram illustrating input data and output data of an electronic device according to one embodiment.

[0021] FIG. 2 is a diagram illustrating a spatial environment for calibrating a camera sensor and a LiDAR sensor according to one embodiment.

[0022] FIG. 3 is a diagram illustrating a calibration process based on image data according to one embodiment.

[0023] FIG. 4 is a diagram illustrating a calibration process based on point cloud data according to one embodiment.

[0024] FIG. 5 is a diagram illustrating a process for generating coordinate system transformation relationship information between a camera and a lidar according to one embodiment.

[0025] FIG. 6 is a schematic block diagram of an electronic device according to one embodiment.

[0026] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Accordingly, actual implementations are not limited to the specific embodiments disclosed, and the scope of this specification includes modifications, equivalents, or substitutions included in the technical concept described by the embodiments.

[0027] Terms such as "first" or "second" may be used to describe various components, but these terms should be interpreted solely for the purpose of distinguishing one component from another. For example, the first component may be named the second component, and similarly, the second component may be named the first component.

[0028] When it is stated that a component is "connected" to another component, it should be understood that it may be directly connected to or coupled with that other component, or that there may be other components in between.

[0029] Singular expressions include plural expressions unless the context clearly indicates otherwise. In this document, phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B or C,” “at least one of A, B and C,” and “at least one of A, B, or C” may each include any one of the items listed together with the corresponding phrase, or all possible combinations thereof. In this specification, terms such as “comprising” or “having” are intended to designate the existence of the described feature, number, step, action, component, part, or combination thereof, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0030] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this specification.

[0031] As used herein, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0032] As used in this document, the term "part" refers to software or hardware components, such as FPGAs or ASICs, and the "part" performs certain roles. However, the meaning of "part" is not limited to software or hardware. The "part" may be configured to reside in an addressable storage medium or configured to operate one or more processors. For example, the "part" may include components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and "parts" may be combined into a smaller number of components and "parts" or further separated into additional components and "parts." Furthermore, the components and "parts" may be implemented to operate one or more CPUs within a device or secure multimedia card. Additionally, '~part' may include one or more processors.

[0033] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. It should be noted that the following embodiments may be referenced, borrowed, or combined with one another. In the description with reference to the accompanying drawings, identical components are given the same reference numeral regardless of the drawing number, and redundant descriptions thereof will be omitted.

[0034]

[0035] FIG. 1 is a diagram illustrating input data and output data of an electronic device according to one embodiment.

[0036] Referring to FIG. 1, according to one embodiment, an electronic device (101) can generate output data (20) based on input data (10). The output data (20) may include calibration data (22) for calibrating a LiDAR sensor (e.g., LiDAR sensor (232) of FIG. 2) and a camera sensor (e.g., camera sensor (234) of FIG. 2) of a vehicle (e.g., vehicle (230) of FIG. 2). The calibration data (22) may include data necessary to align the camera coordinate system and the LiDAR coordinate system. For example, the calibration data (22) may include transformation relationship information that defines the relative position and / or relative direction between the camera sensor and the LiDAR sensor.

[0037] The electronic device (101) can calibrate the camera sensor and the lidar sensor based on the calibration data (22) so that the image data acquired by the camera sensor and the point cloud data acquired by the lidar sensor can be represented on the same coordinate system.

[0038] The input data (10) may include image data (12) and point cloud data (14).

[0039] Image data (12) may be obtained through at least one camera sensor mounted on a vehicle (e.g., camera sensor (234) of FIG. 2).

[0040] Point cloud data (14) may be obtained through at least one lidar sensor mounted on a vehicle (e.g., lidar sensor (232) of FIG. 2).

[0041] In the present disclosure, a calibration process is described based on one camera sensor and one lidar sensor, but it should be noted that this is an example for convenience of explanation. For example, a vehicle may be equipped with a plurality of camera sensors and a plurality of lidar sensors, and an electronic device (101) may calibrate the plurality of camera sensors and the plurality of lidar sensors in parallel (or simultaneously) using image data obtained through the plurality of camera sensors and point cloud data obtained through the plurality of lidar sensors.

[0042]

[0043] FIG. 2 is a diagram illustrating a spatial environment for calibrating a camera sensor and a LiDAR sensor according to one embodiment.

[0044] Referring to FIG. 2, according to one embodiment, a turntable (220) and a robot arm (210) may be used to obtain input data (e.g., input data (10) of FIG. 1) of an electronic device for calibration (e.g., electronic device (101) of FIG. 1).

[0045] The camera sensor (232) can capture a calibration board (30) (e.g., a checkerboard) while the vehicle (230) positioned on the turntable (220) is rotated by the turntable (220).

[0046] Image data obtained through the camera sensor (232) (e.g., image data (12) of FIG. 1) may include a marker pattern image (or pixels representing the marker pattern image) of the calibration board (30).

[0047] Image data may include multiple image frames acquired at different points in time. For example, image data may include a first image frame captured when the turntable (220) is rotated by a first angle (e.g., 5 degrees) and a second image frame captured when the turntable (220) is rotated by a second angle (e.g., 10 degrees).

[0048] A calibration board (30) may be mounted on the robot arm (210). The robot arm (210) may move along a set trajectory (40) while the vehicle (230) is rotated by the turntable (220). The position of the calibration board (30) may change along with the movement of the robot arm (210). For example, when the turntable (220) is rotated by a first angle (e.g., 5 degrees), the first image frame captured includes a first marker pattern image of the calibration board (30) when the robot arm (210) is positioned at a first point (e.g., point (WP1)) of the set trajectory (40), and when the turntable (220) is rotated by a second angle (e.g., 10 degrees), the second image frame captured includes a second marker pattern image of the calibration board (30) when the robot arm (210) is positioned at a second point (e.g., point (WP2)) on the set trajectory (40).

[0049] The LiDAR sensor (234) can acquire point cloud data (e.g., point cloud data (14) of FIG. 1) including points corresponding to the calibration board (30) while the vehicle (230) located on the turntable (220) is rotated by the turntable (220).

[0050] Point cloud data acquired through the lidar sensor (234) may include at least one point cloud frame corresponding to image data acquired through the camera sensor (232). For example, the point cloud data may include a first point cloud frame corresponding to a first image frame captured when the turntable (220) is rotated by a first angle, and a second point cloud frame corresponding to a second image frame captured when the turntable (220) is rotated by a second angle.

[0051]

[0052] FIG. 3 is a diagram illustrating a calibration process based on image data according to one embodiment.

[0053] Referring to FIG. 3, according to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) can estimate the spatial position of a calibration board (e.g., the calibration board (30) of FIG. 2) based on image data (e.g., the image data (12) of FIG. 1) obtained through a camera sensor (e.g., the camera sensor (234) of FIG. 2).

[0054] In operation 310, the electronic device can acquire image data.

[0055] In operation 320, the electronic device can identify corner points (or grid intersections) from marker pattern images (or pixels representing marker pattern images) within an image frame included in the image data.

[0056] In operation 330, the electronic device can reconstruct a three-dimensional plane of the calibration board (30) based on identified corner points. For example, the electronic device can estimate the position and / or orientation of the calibration board (30) in three-dimensional space based on identified corner points.

[0057] In operation 340, the electronic device can estimate a normal vector of the reconstructed three-dimensional plane of the calibration board (30). The estimated normal vector can provide information about the position and / or orientation of the calibration board (30) relative to the camera coordinate system.

[0058]

[0059] FIG. 4 is a diagram illustrating a calibration process based on point cloud data according to one embodiment.

[0060] Referring to FIG. 4, according to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) can estimate the spatial position of a calibration board (e.g., the calibration board (30) of FIG. 2) based on point cloud data (e.g., the point cloud data (14) of FIG. 1) obtained through a lidar sensor (e.g., the lidar sensor (232) of FIG. 2).

[0061] In operation 410, the electronic device can acquire point cloud data.

[0062] In operation 420, the electronic device can extract target data for a region of interest (e.g., the region where the calibration board is located) from a point cloud frame included in the point cloud data.

[0063] In operation 430, the electronic device can estimate the three-dimensional plane of the calibration board from the extracted target data. For example, the electronic device can use the RANSAC (random sample consensus) algorithm to estimate the three-dimensional plane of the calibration board with high accuracy.

[0064] In operation 440, the electronic device can group points (e.g., inliers) that lie to the estimated 3D plane. Through grouping, the electronic device can create a cluster of points representing the calibration board.

[0065] In operation 450, the electronic device can estimate the normal vector of the three-dimensional plane of the calibration board (30) based on the LiDAR sensor coordinate system from the generated cluster.

[0066]

[0067] FIG. 5 is a diagram illustrating a process for generating coordinate system transformation relationship information between a camera and a lidar according to one embodiment.

[0068] Referring to FIG. 5, according to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) can generate calibration data (e.g., calibration data (22) of FIG. 1) based on data providing information about the spatial position of a calibration board (e.g., the calibration board (30) of FIG. 2) obtained through the calibration process of FIG. 3 and the calibration process of FIG. 4.

[0069] In operation 510, the electronic device may perform an initial pose (or initial alignment) between the camera sensor (e.g., camera sensor (234) of FIG. 2) and the lidar sensor (e.g., lidar sensor (232) of FIG. 2). The electronic device may estimate rotation and / or translation for alignment between the camera coordinate system and the lidar coordinate system.

[0070] In operation 520, the electronic device can correct the estimated initial attitude. For example, the electronic device can finely adjust the initial attitude using non-linear optimization.

[0071] In operation 530, the electronic device can generate calibration data for alignment between the LiDAR coordinate system and the camera coordinate system based on the corrected initial attitude. For example, the calibration data may include information on coordinate system transformation relationships between the camera coordinate system and the LiDAR coordinate system. Based on the calibration data, the electronic device can represent (or interpret) image data and point cloud data on a single integrated coordinate system.

[0072]

[0073] FIG. 6 is a schematic block diagram of an electronic device according to one embodiment.

[0074] Referring to FIG. 6, according to one embodiment, the electronic device (101) may include at least one processor (620) and memory (640).

[0075] The memory (640) may store instructions (or programs) executable by at least one processor (620). For example, the instructions may include instructions for executing the operation of at least one processor (620) and / or the operation of each configuration of at least one processor (620).

[0076] The memory (640) may include one or more computer-readable storage media. The memory (640) may include non-volatile storage devices (e.g., magnetic hard disc, optical disc, floppy disc, flash memory, EPROM (electrically programmable memories), EEPROM (electrically erasable and programmable)).

[0077] The memory (640) may be a non-transitory medium. The term “non-transitory” may indicate that the storage medium is not implemented by a carrier wave or a propagated signal. However, the term “non-transitory” should not be interpreted as meaning that the memory (640) is immobile.

[0078] At least one processor (620) can process data stored in memory (640). At least one processor (620) can execute computer-readable code (e.g., software) stored in memory (640) and instructions triggered by at least one processor (620).

[0079] At least one processor (620) may be a data processing device implemented in hardware having a circuit having a physical structure for executing desired operations. For example, the desired operations may include code or instructions included in a program.

[0080] For example, a data processing device implemented in hardware may include a microprocessor, a central processing unit, a processor core, a multi-core processor, a multiprocessor, an Application-Specific Integrated Circuit (ASIC), and a Field Programmable Gate Array (FPGA).

[0081] At least one processor (620) may include a main processor (e.g., a central processing unit or an application processor) and an auxiliary processor (e.g., a communication processor, a neural processing unit (NPU), and / or a graphic processing unit (GPU)).

[0082] At least one processor (620) can enable the electronic device (110) to perform at least one operation by individually or collectively executing code, instructions, and / or applications stored in memory (640).

[0083] The electronic device (101) may further include a communication module (660) as needed. The communication module (660) may establish a direct communication channel (e.g., a wired communication channel) or a wireless communication channel between the electronic device (101) and at least one external device, and may support communication through the established communication channel.

[0084]

[0085] The embodiments described above may be implemented as hardware components, software components, and / or combinations of hardware and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0086] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be stored on any type of machine, component, physical device, virtual equipment, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on computer-readable recording media.

[0087] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may store program instructions, data files, data structures, etc., either individually or in combination, and the program instructions recorded on the medium may be those specifically designed and configured for the embodiment or those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0088] The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0089] Although the embodiments have been described above with reference to the limited drawings, those skilled in the art can apply various technical modifications and variations based thereon. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0090]

[0091] Although the present disclosure has been illustrated and described with reference to various embodiments, it will be understood by those skilled in the art that the various embodiments are intended to be illustrative and not limiting. It will be understood by those skilled in the art that various modifications to the form and details may be made without departing from the true spirit and full scope of the present disclosure, including the appended claims and their equivalents. It will also be understood by those skilled in the art that any of the embodiments described herein may be used in conjunction with other embodiments described herein. Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the claims set forth below.

[0092] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood from this document by those skilled in the art to which the present disclosure belongs.

Claims

1. A device for calibrating a camera sensor and a LiDAR sensor, At least one processor; and memory that stores instructions Includes, When the above instructions are executed individually or collectively by the at least one processor, the device is made to perform a plurality of operations, and The above plurality of operations are, An operation of acquiring image data including a marker pattern image of a calibration board mounted on a robot arm located around the vehicle, through a camera sensor mounted on the vehicle while the vehicle positioned on the turntable is rotated by the turntable; The operation of acquiring point cloud data including points corresponding to the calibration board through the lidar sensor mounted on the vehicle while the vehicle is rotated by the turntable; and Calibrating the camera sensor and the LiDAR sensor based on the image data and the point cloud data. A device including 2. In Paragraph 1, The above image data is, A first image frame captured when the turntable is rotated by a first angle, and a second image frame captured when the turntable is rotated by a second angle. A device including 3. In Paragraph 2, The above point cloud data is, A first point cloud frame corresponding to the first image frame and a second point cloud frame corresponding to the second image frame A device including 4. In Paragraph 2, The above first image frame is, It includes a first marker pattern image of the calibration board when the robot arm is positioned at a first waypoint on a set trajectory, and The above second image frame is, A device comprising a second marker pattern image of the calibration board when the robot arm is positioned at a second point on the set trajectory.

5. In Paragraph 1, The above calibration operation is, An operation to determine a first normal vector of the calibration board based on the above image data; The operation of determining a second normal vector of the calibration board based on the above point cloud data; and Calibrating the camera sensor and the lidar sensor based on the first normal vector and the second normal vector. A device including 6. In Paragraph 1, The above calibration operation is, Operation of simultaneously calibrating multiple camera sensors and multiple LiDAR sensors A device including 7. In Paragraph 4, The above robot arm is, A device that moves along the set trajectory while the vehicle is rotated on the turntable.

8. A method for calibrating a camera sensor and a LiDAR sensor, An operation of acquiring image data including a marker pattern image of a calibration board mounted on a robot arm located around the vehicle, through a camera sensor mounted on the vehicle while the vehicle positioned on the turntable is rotated by the turntable; The operation of acquiring point cloud data including points corresponding to the calibration board through the lidar sensor mounted on the vehicle while the vehicle is rotated by the turntable; and Calibrating the camera sensor and the LiDAR sensor based on the image data and the point cloud data. A method including 9. In Paragraph 8, The above image data is, A first image frame captured when the turntable is rotated by a first angle, and a second image frame captured when the turntable is rotated by a second angle. A method including 10. In Paragraph 9, The above point cloud data is, A first point cloud frame corresponding to the first image frame and a second point cloud frame corresponding to the second image frame A method including 11. In Paragraph 9, The above first image frame is, It includes a first marker pattern image of the calibration board when the robot arm is positioned at a first point on a set trajectory, and The above second image frame is, A method comprising a second marker pattern image of the calibration board when the robot arm is positioned at a second point on the set trajectory.

12. In Paragraph 8, The above calibration operation is, An operation to determine a first normal vector of the calibration board based on the above image data; The operation of determining a second normal vector of the calibration board based on the above point cloud data; and Calibrating the camera sensor and the lidar sensor based on the first normal vector and the second normal vector. A method including 13. In Paragraph 8 The above calibration operation is, Operation of simultaneously calibrating multiple camera sensors and multiple LiDAR sensors A method including 14. In Paragraph 11 The above robot arm is, A method of moving along the set trajectory while the vehicle is rotated on the turntable.

15. A computer program stored on a computer-readable recording medium in combination with hardware to execute the method of any one of claims 8 through 14.

Images