Method and apparatus for generating bird's-eye-view surround view image, and device and program product

Abstract

A method and apparatus for generating a bird's-eye-view surround view image, and a device and a program product, which relate to the technical field of intelligent driving. The method comprises: acquiring images collected by cameras in a vehicle, wherein the images collected by the cameras are combined to completely cover a ground scene around the vehicle; and determining, by means of parallel processing, target pixel values corresponding to original pixel values in collected images, and on the basis of a pixel position mapping relationship, filling a bird's-eye-view surround view image template with the target pixel values, so as to obtain a bird's-eye-view surround view image of the vehicle, wherein the pixel position mapping relationship is a pixel position mapping relationship between the collected images and the bird's-eye-view surround view image template. The problems of a complex generation process and slow generation speed of a bird's-eye-view surround view image are solved.

Description

Bird's-eye view panoramic image generation methods, devices, equipment and program products

[0001] This disclosure claims priority to Chinese patent application No. 202411560876.1, filed on November 4, 2024, entitled “Method, Apparatus, Device and Procedure for Generating Bird’s-eye View Panoramic Images”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of intelligent driving technology, specifically to a method, apparatus, device, and program product for generating bird's-eye view panoramic images. Background Technology

[0003] Bird's-eye view panoramic images play an important role in various intelligent driving scenarios for vehicles. Currently, the process of generating bird's-eye view panoramic images is relatively complex and slow, and further optimization is needed. Summary of the Invention

[0004] This disclosure provides a method, apparatus, device, and program product for generating bird's-eye view panoramic images. The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0005] Firstly, this disclosure provides a method for generating a bird's-eye view panoramic image, including:

[0006] The images captured by each camera in the vehicle are acquired, and the images captured by each camera are combined to completely cover the ground scene around the vehicle.

[0007] The target pixel value corresponding to the original pixel value in the acquired image is determined by parallel processing, and the target pixel value is filled into the bird's-eye view panoramic image template according to the pixel position mapping relationship to obtain the bird's-eye view panoramic image of the vehicle.

[0008] The pixel position mapping relationship refers to the mapping relationship between the pixel positions of the acquired image and the bird's-eye view panoramic image template.

[0009] Secondly, this disclosure provides a bird's-eye view panoramic image generation device, comprising:

[0010] The acquisition module is configured to acquire images from various cameras in the vehicle, wherein the images from each camera are combined to completely cover the ground scene around the vehicle.

[0011] The processing module is configured to determine the target pixel value corresponding to the original pixel value in the acquired image through parallel processing, and fill the target pixel value into the bird's-eye view panoramic image template according to the pixel position mapping relationship to obtain the bird's-eye view panoramic image of the vehicle.

[0012] The pixel position mapping relationship refers to the mapping relationship between the pixel positions of the acquired image and the bird's-eye view panoramic image template.

[0013] Thirdly, this disclosure provides an electronic device, including:

[0014] At least one processor; and

[0015] A memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores at least one computer program that can be executed by the at least one processor, the at least one computer program being executed by the at least one processor to enable the at least one processor to perform the bird's-eye view panoramic image generation method as described in the first aspect.

[0017] Fourthly, this disclosure provides a computer program product, which includes a computer program that, when run in a processor, implements the bird's-eye view panoramic image generation method described in the first aspect.

[0018] The embodiments provided in this disclosure predetermine the pixel position mapping relationship between the acquired images and the bird's-eye view panoramic image template. During the generation of the bird's-eye view panoramic image, cameras within the vehicle capture the ground scene around the vehicle, combining the images from each camera to completely cover the ground scene around the vehicle. Parallel processing is used to determine the target pixel values ​​corresponding to the original pixel values ​​in the acquired images, and these target pixel values ​​are then filled into the bird's-eye view panoramic image template according to the pixel position mapping relationship, resulting in the bird's-eye view panoramic image of the vehicle. The bird's-eye view panoramic image can be generated using images from multiple cameras and a predetermined pixel position mapping relationship, simplifying the generation process. Furthermore, parallel processing improves the generation efficiency of the bird's-eye view panoramic image. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0020] Figure 1 is a schematic diagram of the method for generating a bird's-eye panoramic view in an embodiment of this disclosure;

[0021] Figure 2 shows a schematic diagram of the visible range of the four-channel fisheye camera in an embodiment of this disclosure;

[0022] Figure 3 shows a schematic diagram of the placement of the target and the vehicle in an embodiment of this disclosure;

[0023] Figure 4 is a schematic diagram of the process of calibrating the external parameters of a fisheye camera in an embodiment of this disclosure.

[0024] Figure 5 shows a schematic diagram of the pixel position mapping relationship from a fisheye image to a bird's-eye panoramic image template in an embodiment of this disclosure;

[0025] Figure 6 is a schematic diagram of the effect of stitching together a bird's-eye view panoramic image in an embodiment of this disclosure;

[0026] Figure 7 shows a schematic diagram of the correspondence between each region and each camera in the bird's-eye view panoramic view of this embodiment;

[0027] Figure 8 is a block diagram of the bird's-eye view panoramic image generation device in an embodiment of this disclosure;

[0028] Figure 9 shows a schematic diagram of the structure of the electronic device in an embodiment of this disclosure. Detailed Implementation

[0029] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0030] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0031] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0032] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0033] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0034] Overview

[0035] The existing process for generating a vehicle bird's-eye view panoramic image requires selecting a calibration area and marking it with line lasers to form a rectangular laser area. Cameras are installed at eight positions on the vehicle: front left, front right, rear left, rear right, front, rear, left, and right. The total field of view of the eight cameras covers 360 degrees around the vehicle, with some overlap between adjacent cameras. The vehicle is then moved to the center of the laser area, establishing a vehicle coordinate system. A checkerboard pattern is placed within the camera's field of view, and images of the checkerboard pattern within the camera's field of view are captured. The coordinates of the upper corner points of the checkerboard pattern in the vehicle coordinate system are calculated. The coordinates of the lower corner points in the image coordinate system are also calculated. The projection transformation matrix corresponding to the checkerboard pattern image is calculated, converting the checkerboard pattern image into a bird's-eye view image. Finally, all the bird's-eye view images are stitched together to obtain the vehicle bird's-eye view panoramic image.

[0036] The current bird's-eye view panoramic image generation process relies on lasers and requires up to eight cameras, making it complex and slow in stitching. Therefore, this disclosure provides a method for generating bird's-eye view panoramic images that eliminates the need for lasers, reduces the number of cameras, and improves generation efficiency.

[0037] Exemplary methods

[0038] This disclosure provides a method for generating a bird's-eye view panoramic image. This method can be used in a vehicle or in an electronic device that establishes communication with the vehicle, as shown in Figure 1. The method mainly includes the following steps:

[0039] Step 101: Acquire images from various cameras in the vehicle, and combine the images from various cameras to completely cover the ground scene around the vehicle.

[0040] In some embodiments, each camera includes four fisheye cameras: a front-view fisheye camera, a left-view fisheye camera, a right-view fisheye camera, and a rear-view fisheye camera. Acquiring images from each camera in the vehicle includes: acquiring a front-view fisheye image acquired by the front-view fisheye camera, a left-view fisheye image acquired by the left-view fisheye camera, a right-view fisheye image acquired by the right-view fisheye camera, and a rear-view fisheye image acquired by the rear-view fisheye camera.

[0041] The combination of the front-view, left-view, right-view, and rear-view fisheye images completely covers the ground scene around the vehicle, achieving 360° full coverage. Figure 2 shows a schematic diagram of the field of view of the four fisheye cameras. The four fisheye images are stitched together based on the overlapping parts of the images acquired by the four cameras to obtain a captured image that fully covers the area around the vehicle.

[0042] By using four fisheye cameras, 360° full coverage of the scene around the vehicle can be achieved, reducing the number of cameras, reducing implementation complexity, and saving costs.

[0043] Step 102: Determine the target pixel value corresponding to the original pixel value in the acquired image through parallel processing, and fill the bird's-eye view panoramic image template with the target pixel value according to the pixel position mapping relationship to obtain the bird's-eye view panoramic image of the vehicle; wherein, the pixel position mapping relationship is the mapping relationship between the pixel positions of the acquired image and the bird's-eye view panoramic image template.

[0044] The embodiments disclosed herein do not limit the specific processing procedures and methods of parallel processing. By using parallel processing to convert the original pixel values ​​in the acquired image into the corresponding pixel values ​​in the bird's-eye view panoramic image template, the processing efficiency can be significantly improved compared to the serial processing method.

[0045] For example, the parallel processing procedure is as follows:

[0046] The pixel position mapping relationship is configured in the parallel computing kernel function of the graphics processing unit (GPU) using the OpenCL (Open Computing Language).

[0047] The front-view fisheye image, left-view fisheye image, right-view fisheye image, and rear-view fisheye image in NV12 format, along with the bird's-eye panoramic image template, are passed into the parallel computing kernel function.

[0048] The parallel computing kernel function iterates through each pixel of each fisheye image in parallel, calculates the R, G, B values ​​of the pixel and its offset in the bird's-eye panoramic image template, and assigns values ​​to the corresponding pixels in the bird's-eye panoramic image template.

[0049] If the fisheye image format is another format, then apply the corresponding pixel conversion formula.

[0050] Furthermore, the parallel computing kernel function can also incorporate subsequent processing of the bird's-eye view panoramic image, such as quantization and normalization.

[0051] In some embodiments, the process of obtaining the pixel position mapping relationship includes: determining the extrinsic parameters of each camera by calibrating the scene; wherein the calibration scene is a target placed around the vehicle; and determining the pixel position mapping relationship between the acquired image and the bird's-eye view panoramic image template based on the extrinsic and intrinsic parameters of each camera.

[0052] In this calibration scenario, the targets are pre-positioned around the vehicle, corresponding to the positions of the vehicle's various fisheye cameras. The origin of the world coordinate system is pre-set, and the center of the vehicle's rear axle is aligned with the origin of the world coordinate system. The placement of the targets and the vehicle is shown in Figure 3. Here, a checkerboard calibration board is used as an example; however, it is not mandatory to use a checkerboard pattern calibration board, and other patterns can also be used.

[0053] In some embodiments, determining the extrinsic parameters of each camera through scene calibration includes: acquiring calibration scene images collected by each camera, wherein the combined calibration scene images of each camera can completely cover the targets placed around the vehicle; for each camera, determining the camera's extrinsic parameters based on the camera's intrinsic parameters, the relative positions of the target's corner points and the vehicle, and the pixel coordinates of the corner points in the calibration scene images. This method can improve the calibration accuracy of camera extrinsic parameters, providing a basis for ensuring the accuracy of the bird's-eye view panoramic image, and the calibration process is simple and quick.

[0054] Specifically, the relative position of the target's corner point to the vehicle is the relative position of the target's corner point to the origin of the world coordinate system.

[0055] In some embodiments, the extrinsic parameters of the camera are determined based on the camera's intrinsic parameters, the relative position of the target's corner point and the vehicle, and the pixel coordinates of the corner point in the calibration scene image. This includes: determining the world coordinates of the corner point in the world coordinate system based on the relative position of the target's corner point and the vehicle, wherein the origin of the world coordinate system is the rear axle center point of the vehicle, the X-axis is parallel to the ground and points in front of the vehicle, the Y-axis is parallel to the ground and points to the left side of the vehicle, and the Z-axis is perpendicular to the ground and points upward; and the extrinsic parameters of the camera are determined based on the camera's intrinsic parameters, the pixel coordinates of the corner point, and the world coordinates.

[0056] In some embodiments, determining the camera's extrinsic parameters based on the camera's intrinsic parameters, the pixel coordinates corresponding to the corner points, and the world coordinates includes: determining the camera's initial extrinsic parameters based on the camera's intrinsic parameters, the pixel coordinates corresponding to the corner points, and the world coordinates; determining the world coordinate verification value corresponding to the corner points based on the initial extrinsic parameters, the camera's intrinsic parameters, and the pixel coordinates corresponding to the corner points; and optimizing the camera's initial extrinsic parameters based on the difference between the world coordinates and the world coordinate verification value.

[0057] In some embodiments, acquiring calibration scene images captured by each camera includes: acquiring a front-view calibration scene image captured by a front-view fisheye camera, a left-view calibration scene image captured by a left-view fisheye camera, a right-view calibration scene image captured by a right-view fisheye camera, and a rear-view calibration scene image captured by a rear-view fisheye camera.

[0058] In an exemplary embodiment, Figure 4 illustrates the process of calibrating the extrinsic parameters of a fisheye camera. The specific process is as follows:

[0059] Step 401: Arrange the target on a flat ground according to the design dimensions and relative position.

[0060] Step 402: Park the vehicle equipped with four fisheye cameras (front, rear, left, and right) in the middle of the target set up on the ground.

[0061] Specifically, the targets are divided into four sections: front, rear, left, and right. These sections are laid out on a flat surface according to the relative positions shown in Figure 3. Each target section has a uniformly sized, black-and-white checkerboard pattern, with preset dimensions. The relative positions between the targets are fixed. A vehicle equipped with four fisheye cameras (front, rear, left, and right) is parked within the central blank area enclosed by the four targets.

[0062] Step 403: Acquire images from the four fisheye cameras and calculate the pixel coordinates of each corner of the target in the acquired images.

[0063] Specifically, after the site is calibrated and set up, images containing the target pattern are collected from four fisheye cameras (front, back, left, and right) at the current moment.

[0064] In image processing, the top-left corner of the image is defined as the origin of the pixel coordinate system, the direction from the top-left corner to the top-right corner is defined as the positive direction of the u-axis, and the direction from the top-left corner to the bottom-left corner is defined as the positive direction of the v-axis. Calculating the pixel coordinates of the corner points involves using image processing algorithms to calculate the pixel coordinate values ​​of the target pattern's corner points in the pixel coordinate system.

[0065] Step 404: Measure the world coordinates of the target in the world coordinate system.

[0066] The origin of the world coordinate system is the center point of the rear axle of the vehicle. The positive directions of its X and Y axes are shown in Figure 3, and the positive direction of its Z axis is defined as perpendicular to the ground and upwards. Calculating the world coordinate values ​​of corner points involves using the known dimensions of the target pattern to calculate the coordinate values ​​of each corner point in the world coordinate system.

[0067] Step 405: Obtain the camera's intrinsic parameters and calculate the distortion-free pixel coordinates corresponding to the corner points of the target.

[0068] By using the camera's intrinsic parameters and the pixel coordinates of the corner points in the target pattern, the pixel coordinates of the corresponding corner points in the distortion-free image can be calculated.

[0069] Step 406: Calculate the initial extrinsic parameters of the camera using the camera's intrinsic parameters, the pixel coordinates (i.e., the undistorted pixel coordinates) of the corner points in the target pattern, and the world coordinates.

[0070] Step 407: Optimize the initial extrinsic parameters by minimizing the backprojection error to obtain highly accurate extrinsic parameters.

[0071] Specifically, camera extrinsic parameters are expressed as the camera's pose in the world coordinate system, and can be represented by a rotation matrix and a translation vector. The principle of projecting a point in the world coordinate system onto the image plane can be expressed by the following formula. Using the camera intrinsic parameters and the pixel coordinates of the target pattern's corner points, the pixel coordinates of the corresponding corner points in the distortion-free image can be calculated. Knowing the distortion-free pixel coordinates of the target corner points, the world coordinates, and the camera intrinsic parameters, the camera's extrinsic parameters can be obtained. Then, by minimizing the backprojection errors of all corner points, highly accurate extrinsic parameters are optimized.

[0072] The coordinates in the pixel coordinate system are represented as (u,v), and the coordinates in the world coordinate system are represented as (Xw, Yw, Zw). R is the rotation matrix, T is the translation matrix, fx and fy are the focal lengths in the x and y directions, respectively, and are usually the same. cx and cy are the coordinates of the optical center in the pixel coordinate system, respectively, and Zc represents the depth value from the camera's imaging plane.

[0073] The process involves optimizing the initial extrinsic parameters by minimizing the backprojection error to obtain highly accurate extrinsic parameters. The specific steps are as follows: The pixel coordinates of the target corner points in the image are already determined. Using the intrinsic parameters, pixel coordinates, and the initially calculated extrinsic parameters, the calculated world coordinates of the target corner points can be obtained. The calculated values ​​differ from the theoretical values. An objective function is constructed using the coordinate differences of all target corner points, where the parameters of the objective function are the extrinsic parameters. The corresponding extrinsic parameters are obtained by minimizing the objective function; these are the highly accurate extrinsic parameter values.

[0074] In some embodiments, determining the pixel position mapping relationship between the acquired image and the bird's-eye view panoramic image template based on the extrinsic and intrinsic parameters of each camera includes: determining the world coordinate points of the pixels in the bird's-eye view panoramic image template; determining the camera coordinate points of the camera's camera coordinate system based on the camera's intrinsic and extrinsic parameters; determining the pixel coordinate points of the acquired image based on the pixel points of the acquired image; and determining the pixel position mapping relationship between the acquired image and the bird's-eye view panoramic image template based on the pixels in the bird's-eye view panoramic image template corresponding to the pixels in the acquired image.

[0075] For example, taking a fisheye image as the acquired image, Figure 5 shows a schematic diagram of the pixel position mapping relationship from a fisheye image to a bird's-eye panoramic (BEV) template. The mapping process is as follows:

[0076] a. Pixel coordinate system (i,j) on BEV -> b. World coordinate system (x,y) -> c. Camera coordinate system (x,y,z) -> d. Fisheye image coordinate system (x,y); During this process, BEV can be regarded as a 640*640 blank table;

[0077] Here, a->b is to calculate the real-world coordinates of each cell in the BEV template (i.e., the position of the pixel to be filled in the BEV template). The origin of the real-world coordinates is the center of the rear axle of the vehicle. The field of view of the entire BEV is 16m*16m. Since the size of the final BEV template is known, each pixel on the BEV corresponds to a position (x, y) in the world coordinate system.

[0078] b->c maps points in the world coordinate system to those in the fisheye camera based on the camera's intrinsic and extrinsic parameters. z = 1 is due to the characteristics of the fisheye camera.

[0079] c->d obtains the corresponding position on the fisheye image from the point in the camera coordinate system based on the distortion correction parameters, that is, the pixel position on the fisheye image. The offset is used because the pixel value is determined by the offset in storage.

[0080] This completes the pixel position lookup table (i.e., LUT table) from the BEV template to the fisheye image, thus obtaining the corresponding pixel position mapping relationship.

[0081] Figure 6 shows a schematic diagram of the bird's-eye panoramic image created by stitching together fisheye images captured by four fisheye cameras in a calibration scene according to the pixel position mapping relationship. The correspondence between each region and each camera in the stitched bird's-eye panoramic image is shown in Figure 7.

[0082] The embodiments provided in this disclosure predetermine the pixel position mapping relationship between the acquired images and the bird's-eye view panoramic image template. During the generation of the bird's-eye view panoramic image, cameras within the vehicle capture the ground scene around the vehicle, combining the images from each camera to completely cover the ground scene around the vehicle. Parallel processing determines the target pixel values ​​corresponding to the original pixel values ​​in the acquired images, and these target pixel values ​​are filled into the bird's-eye view panoramic image template according to the pixel position mapping relationship, resulting in the bird's-eye view panoramic image of the vehicle. The bird's-eye view panoramic image can be generated using images from multiple cameras and a predetermined pixel position mapping relationship, simplifying the generation process. Furthermore, parallel processing improves the generation efficiency of the bird's-eye view panoramic image.

[0083] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic, and the execution order between steps is not limited to implementation according to step number.

[0084] Exemplary device

[0085] In addition, this disclosure also provides a bird's-eye view panoramic image generation device, electronic device, and computer program product, all of which can be used to implement any of the bird's-eye view panoramic image generation methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding records in the method section and will not be repeated here.

[0086] Figure 8 is a block diagram of a bird's-eye view panoramic image generation device provided in an embodiment of this disclosure. The device mainly includes:

[0087] The acquisition module 801 is configured to acquire images from various cameras in the vehicle, wherein the images from various cameras are combined to completely cover the ground scene around the vehicle.

[0088] The processing module 802 is configured to determine the target pixel value corresponding to the original pixel value in the acquired image through parallel processing, and fill the target pixel value into the bird's-eye view panoramic image template according to the pixel position mapping relationship to obtain the bird's-eye view panoramic image of the vehicle.

[0089] The pixel position mapping relationship refers to the mapping relationship between the pixel positions of the acquired image and the bird's-eye view panoramic image template.

[0090] Each module in the above-mentioned device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0091] Exemplary device

[0092] Figure 9 is a block diagram of an electronic device provided in an embodiment of this disclosure.

[0093] Referring to FIG9, an embodiment of this disclosure provides an electronic device, which includes: at least one processor 901; at least one memory 902; and one or more input / output interfaces (I / O interfaces) 903 connected between the processor 901 and the memory 902; wherein, the memory 902 stores one or more computer programs that can be executed by at least one processor 901, and the one or more computer programs are executed by at least one processor 901 to enable at least one processor 901 to perform the above-described bird's-eye view panoramic image generation method.

[0094] The modules in the aforementioned electronic devices can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0095] Exemplary computer program products and storage media

[0096] This disclosure also provides a computer program product, including a computer program that, when run in a processor, implements the above-described bird's-eye view panoramic image generation method.

[0097] Computer programs can be stored on readable storage media of a computer device or in the cloud; the processor of a computer device reads computer programs from readable storage media or in the cloud.

[0098] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically manifested as a computer storage medium; in another optional embodiment, the computer program product is specifically manifested as a software product, such as a software development kit (SDK), etc.

[0099] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0100] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0101] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0102] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0103] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0104] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0105] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0106] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0107] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0108] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications or equivalent substitutions made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for generating a bird's-eye view panoramic image, comprising: The images captured by each camera in the vehicle are acquired, and the images captured by each camera are combined to completely cover the ground scene around the vehicle. The target pixel value corresponding to the original pixel value in the acquired image is determined by parallel processing, and the target pixel value is filled into the bird's-eye view panoramic image template according to the pixel position mapping relationship to obtain the bird's-eye view panoramic image of the vehicle. The pixel position mapping relationship refers to the mapping relationship between the pixel positions of the acquired image and the bird's-eye view panoramic image template.

2. The method according to claim 1, wherein, The process of obtaining the pixel position mapping relationship includes: The extrinsic parameters of each camera are determined by calibrating a scene, wherein the calibration scene is a target placed around the vehicle. Based on the extrinsic and intrinsic parameters of each camera, the pixel position mapping relationship between the acquired image and the bird's-eye view panoramic image template is determined.

3. The method according to claim 2, wherein, The process of determining the extrinsic parameters of each camera through scene calibration includes: The calibration scene images captured by each of the cameras are acquired, and the calibration scene images of each camera are combined to completely cover the targets placed around the vehicle. For each camera, the extrinsic parameters of the camera are determined based on the camera's intrinsic parameters, the relative position of the target's corner point to the vehicle, and the pixel coordinates of the corner point in the calibration scene image.

4. The method according to any one of claims 1 to 3, wherein, The cameras mentioned above include four fisheye cameras: a front-view fisheye camera, a left-view fisheye camera, a right-view fisheye camera, and a rear-view fisheye camera. The acquisition of calibration scene images captured by each of the cameras includes: Acquire the front-view calibration scene image captured by the front-view fisheye camera, the left-view calibration scene image captured by the left-view fisheye camera, the right-view calibration scene image captured by the right-view fisheye camera, and the rear-view calibration scene image captured by the rear-view fisheye camera.

5. The method according to claim 3 or 4, wherein, The step of determining the extrinsic parameters of the camera based on the camera's intrinsic parameters, the relative position of the target's corner points to the vehicle, and the pixel coordinates of the corner points in the calibration scene image includes: Based on the relative position of the corner point of the target and the vehicle, the world coordinates of the corner point in the world coordinate system are determined. The origin of the world coordinate system is the center point of the rear axle of the vehicle, the X-axis is parallel to the ground and points in front of the vehicle, the Y-axis is parallel to the ground and points to the left side of the vehicle, and the Z-axis is perpendicular to the ground and faces upward. The extrinsic parameters of the camera are determined based on the camera's intrinsic parameters, the pixel coordinates corresponding to the corner points, and the world coordinates.

6. The method according to claim 5, wherein, Determining the extrinsic parameters of the camera based on its intrinsic parameters, the pixel coordinates corresponding to the corner points, and the world coordinates includes: The initial extrinsic parameters of the camera are determined based on the camera's intrinsic parameters, the pixel coordinates corresponding to the corner points, and the world coordinates. Based on the initial extrinsic parameters, the camera's intrinsic parameters, and the pixel coordinate values ​​corresponding to the corner points, determine the world coordinate verification value corresponding to the corner points; The initial extrinsic parameters of the camera are optimized based on the difference between the world coordinate value and the world coordinate verification value.

7. The method according to any one of claims 2 to 6, wherein, The step of determining the pixel position mapping relationship between the acquired image and the bird's-eye view panoramic image template based on the extrinsic and intrinsic parameters of each camera includes: Determine the mapping of pixels in the bird's-eye view panoramic image template to world coordinate points in the world coordinate system; Based on the camera's intrinsic and extrinsic parameters, determine the camera coordinates that map the world coordinates to the camera's camera coordinate system. Determine the mapping of the camera coordinate points to the pixel coordinate points of the acquired image; Based on the pixels in the bird's-eye view template corresponding to the pixels in the acquired image, the pixel position mapping relationship between the acquired image and the bird's-eye view template is determined.

8. A bird's-eye view panoramic image generation device, comprising: The acquisition module is configured to acquire images from various cameras in the vehicle, wherein the images from each camera are combined to completely cover the ground scene around the vehicle. The processing module is configured to determine the target pixel value corresponding to the original pixel value in the acquired image through parallel processing, and fill the target pixel value into the bird's-eye view panoramic image template according to the pixel position mapping relationship to obtain the bird's-eye view panoramic image of the vehicle. The pixel position mapping relationship refers to the mapping relationship between the pixel positions of the acquired image and the bird's-eye view panoramic image template.

9. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores at least one computer program that can be executed by the at least one processor, the at least one computer program being executed by the at least one processor to enable the at least one processor to perform the bird's-eye view panoramic image generation method as described in any one of claims 1-7.

10. A computer program product, wherein, The computer program product includes a computer program that, when run in a processor, implements the bird's-eye view panoramic image generation method as described in any one of claims 1-7.

Images