Surround-view image generation method and apparatus, and vehicle

Abstract

A surround-view image generation method and apparatus, and a vehicle, which relate to the technical field of vehicles. The method comprises: on the basis of state information of a vehicle-mounted first camera, determining a surround-view image blind area of a vehicle, wherein the surround-view image blind area is located within a surround-view collection field of view of the first camera, and the state information of the first camera is used for indicating a pose change of the first camera or an operational fault of the first camera; and on the basis of a first image, generating a first surround-view image of the vehicle at a first moment, wherein the first image comprises an image collected by the first camera prior to the first moment and / or an image collected by at least one second camera, and the first image is used for complementing the surround-view image blind area. The method can ensure the integrity and accuracy of a surround-view image when the state of a vehicle-mounted camera changes, thereby enhancing the safety and convenience of driving.

Description

A method, apparatus and vehicle for generating a surround view image

[0001] This application claims priority to Chinese Patent Application No. 202411644864.7, filed on November 15, 2024, entitled "A Method, Apparatus and Vehicle for Generating a Surround View Image", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of vehicle technology, and in particular to a method, apparatus and vehicle for generating surround view images. Background Technology

[0003] In modern automotive technology, panoramic surround view systems have become an important feature for enhancing driving safety and convenience. This system uses multiple fisheye cameras to capture images of the vehicle's surroundings and processes them using algorithms to generate a seamlessly stitched panoramic view, providing the driver with intuitive information about the vehicle's environment.

[0004] However, in practical applications, especially when the camera's status changes, the generated surround-view image may be incomplete due to changes in external parameters or poor operating conditions. For example, in scenarios such as parking in extremely narrow spaces or passing through narrow passages, drivers often fold down the side mirrors to avoid scratching them. This action changes the status of the side-view cameras mounted on the mirrors, resulting in blind spots on both sides of the panoramic surround-view image. Specifically, these blind spots may have been filled by images captured by the cameras mounted on the mirrors, but the change in camera status prevents the acquisition of accurate image data. Furthermore, when the rearview camera is mounted on the trunk lid, opening the trunk will also affect the accuracy of the surround-view image. Alternatively, if the onboard cameras involved in generating the surround-view image malfunction, the image may also be incomplete or incorrect.

[0005] These technical challenges not only reduce the accuracy and usability of panoramic surround view images, but may also pose a potential threat to vehicle driving safety. Summary of the Invention

[0006] This application discloses a method, apparatus, and vehicle for generating surround view images, which can avoid the problem of incomplete or incorrect surround view image display, especially when the status of the vehicle camera changes, thereby improving the completeness and accuracy of the surround view image display and enhancing driving safety and convenience.

[0007] Firstly, a method for generating surround-view images is disclosed. This method can be applied to a vehicle or to a module (e.g., a chip) within the vehicle. The vehicle may include at least one camera for acquiring images of the vehicle's surroundings. The at least one camera includes a first camera. The following description uses an application to a vehicle as an example. The surround-view image generation method may include:

[0008] Based on the status information of the first camera, the blind spot of the vehicle's surround view image is determined. The blind spot of the surround view image is located within the surround view acquisition field of the first camera. The status information of the first camera is used to indicate the pose change of the first camera or the malfunction of the first camera.

[0009] Based on the first image, a first surround view image of the vehicle at a first moment is generated. The first image includes an image captured by a first camera before the first moment and / or an image captured by at least one second camera. The first image is used to fill in the blind spots of the surround view image. At least one camera also includes a second camera.

[0010] Based on the above solution, it can be seen that this application can intelligently determine the blind spots of the surround view image based on the status information of the first camera on the vehicle. The innovation lies in the fact that it does not only rely on image data at a single moment, but also combines the status information of the camera (such as pose changes or operational malfunctions), thereby enabling more accurate identification of blind spots caused by changes in camera status.

[0011] Furthermore, considering that the images captured by the first camera at a historical moment may include blind spot images of the surround view when the vehicle is in its current position, and the images captured by the second camera at a historical moment or the current moment may also include blind spot images of the surround view when the vehicle is in its current position, by using the historical images of the first camera (such as images acquired before the first moment) and / or the images captured by the second camera at the current or historical moment to supplement the aforementioned blind spot images, the completeness and accuracy of the surround view images are significantly improved when the state of the first camera changes. This provides the driver with a clearer and blind spot-free view of the vehicle's surrounding environment, enhancing driving safety and convenience.

[0012] In some cases, the first camera is a camera involved in generating a surround view image of the vehicle. For example, the cameras involved in generating a surround view image of the vehicle include multiple cameras distributed around the vehicle to obtain images of the vehicle's surroundings. The first camera can be any one or more of the aforementioned multiple cameras. For example, the first camera can be a camera mounted on a rearview mirror facing one side of the vehicle body, or a camera mounted on the trunk lid facing the rear of the vehicle.

[0013] Optionally, the second camera can refer to a camera that does not participate in the generation of the vehicle's surround view image. For example, the second camera can be a camera mounted on the side of the vehicle body facing the front and rear directions of the vehicle body. Alternatively, it can refer to a camera that participates in the generation of the vehicle's surround view image. For example, the second camera can also be a camera facing the front or rear of the vehicle.

[0014] Optionally, to suit different applications, the first or second camera may use a variety of lenses, such as zoom lenses, fixed-focus lenses, fisheye lenses, wide-angle lenses, telephoto lenses, macro lenses, pinhole lenses, etc. This application does not limit the types of lenses used.

[0015] Optionally, the first camera malfunction is used to characterize the first camera's inability to capture an image or incomplete image capture. Optionally, the first moment can refer to the current moment, which can be the moment corresponding to the state information of the first camera. For example, the current moment can be the moment when the pose of the first camera changes, or the moment when the first camera malfunctions.

[0016] Optionally, the blind spot in the surround view image can be the missing part in the surround view image in the case where the surround view image is not fully displayed due to the status information of the first camera.

[0017] For example, the first image obtained above can be an image directly captured by an onboard camera (such as at least one camera mentioned above), or it can be an image obtained by performing image preprocessing operations after being captured by the camera. These image preprocessing operations include image flipping, image rotation, Gaussian filtering, erosion dilation, etc. Optionally, the first surround view image is the vehicle surround view image corresponding to the first moment.

[0018] To fully utilize resources, in an optional implementation, in scenarios where the pose of the first camera changes, the image captured by the first camera at the moment of pose change can still be used. The image after the pose change is combined with the image before the pose change to fill in the blind spots of the surround view image and regenerate the surround view image of the vehicle. Specifically, the first image includes images captured by the first camera before the first moment, and the state information of the first camera is used to indicate the change in the pose of the first camera. Based on the first image, a first surround view image of the vehicle at the first moment is generated, including:

[0019] Based on the first image and the second image, a first panoramic image is generated at a first moment. The second image includes the image captured by the first camera at the first moment, and the field of view corresponding to the second image includes at least a portion of the blind zone of the panoramic image.

[0020] Based on the above scheme, this embodiment further refines the process of generating the panoramic image. Especially in cases of camera pose changes, it combines images captured by the first camera before and after the first moment to generate a more accurate panoramic image. This method not only considers the impact of camera pose changes on the field of view but also fully utilizes historical image data, thereby more accurately filling in blind spots and avoiding image stitching errors caused by camera pose changes. This improvement further enhances the accuracy and practicality of the panoramic image.

[0021] Considering that surround view images are more commonly used in scenarios involving vehicle movement, such as parking or approaching other vehicles, and that this solution is also applicable to similar scenarios, in one optional implementation, based on the first image, generating a first surround view image of the vehicle at a first moment includes:

[0022] While the vehicle remains in motion, a first surround view image is generated based on the first image at a first moment.

[0023] As can be seen from the above solution, this embodiment emphasizes the application advantages of this application when the vehicle is in motion. During vehicle operation, the real-time performance and accuracy of the surround view image are crucial. This application generates the surround view image based on a first image (including historical images and other camera images), ensuring a clear, blind-spot-free surround view image even when the vehicle is in motion.

[0024] To further improve the clarity and accuracy of the generated surround view image, by combining historical images closest to the current moment (first moment) and performing image transformation, a surround view image of the vehicle body at its current position can be generated. In one optional implementation, the first image includes images captured by the first camera before the first moment. Based on the first image, a first surround view image of the vehicle at the first moment is generated, including:

[0025] Based on the third image, a first panoramic image is generated at a first moment. The first image includes the third image, which is an image captured by the first camera at a second moment. The second moment is earlier than the first moment and is closest to the first moment. The field of view corresponding to the third image includes the blind zone of the panoramic image.

[0026] Based on the above scheme, this implementation method emphasizes selecting the historical image (third image) closest to the first moment to fill in the blind spot when using historical image data. By reducing the image time span, it not only reduces the amount of calculation, but also improves the accuracy and timeliness of the surround view image. This feature enables the surround view image to more accurately reflect the actual situation around the vehicle, providing the driver with more reliable driving assistance information.

[0027] In some implementation scenarios, after the state information of the first camera changes, the image it acquires may not include relevant images of the blind spots in the surround view image of the vehicle's current location. For example, in a parking scenario, the first camera is a camera mounted on the trunk lid and facing the rear of the vehicle. Therefore, the first image needs to be refined to ensure that it includes relevant images of the blind spots in the surround view image of the vehicle's current location. In an optional implementation, the first image includes images captured by the first camera before a first moment. Based on the first image, a first surround view image of the vehicle at the first moment is generated, including:

[0028] Based on the fourth image, a first panoramic image at a first moment is generated. The first image includes the fourth image, which is an image captured by the first camera at a third moment. The third moment is earlier than the moment indicated by the state information of the first camera. The field of view corresponding to the fourth image includes the blind zone of the panoramic image.

[0029] Based on the above scheme, this embodiment focuses on generating a surround view image from images captured before the camera's state changes (the fourth image). This method is particularly suitable for situations where the field of view is limited due to changes in the camera's state, such as when the trunk is opened, causing a change in the pose of the first camera. By utilizing image data before the state change, this application can more accurately predict and fill in blind spots, thereby improving the completeness and accuracy of the surround view image.

[0030] In one alternative implementation, generating a first surround view image of the vehicle at a first moment based on the first image includes:

[0031] Determine the position of the first pixel of the first target object in the first image;

[0032] Based on the vehicle's driving status information and the position of the first pixel, the position of the second pixel in the first surround view image of the first target object at the first moment is estimated. The driving status information includes driving distance or driving angle.

[0033] Based on the position of the second pixel, a first surround view image of the vehicle at the first moment is generated.

[0034] Based on the above scheme, it can be seen that by combining the vehicle's driving distance, driving angle and other state information, as well as the target object's position information in the image, this embodiment can more accurately predict the position of the target object in the surround view image.

[0035] In one alternative implementation, before determining the vehicle's surround-view blind spot based on the state information of the first camera, the method further includes:

[0036] Obtain the status information of the first camera.

[0037] As can be seen from the above scheme, obtaining the state information of the first camera is a prerequisite for realizing the panoramic image generation method of this application. By obtaining the camera's state information, this application can intelligently identify blind spots caused by changes in camera state and take corresponding measures to fill in these blind spots. This step ensures the reliability and accuracy of the panoramic image generation method, providing strong support for subsequent image processing and stitching.

[0038] In one optional implementation, the state information of the first camera is used to indicate changes in the pose of the first camera. Obtaining the state information of the first camera includes:

[0039] The system receives a first instruction message, which is used to instruct the folding of the vehicle's rearview mirror. The first camera includes a side-view camera mounted on the vehicle's rearview mirror.

[0040] Based on the above solution, this embodiment is specifically designed for the common scenario of rearview mirror folding. By receiving the first instruction information indicating that the rearview mirror should be folded, this application can accurately identify the blind spots in the surround view image caused by the folding of the rearview mirror and take corresponding measures to fill these blind spots. For example, by using the extrinsic parameters of the first camera before and after the rearview mirror is folded, the blind spots caused by the folding of the rearview mirror can be accurately determined and identified.

[0041] In one optional implementation, the state information of the first camera is used to indicate changes in the pose of the first camera. Obtaining the state information of the first camera includes:

[0042] The system receives a second instruction message, which is used to instruct the opening of the vehicle's trunk. The first camera includes a rearview camera mounted on the vehicle's trunk lid.

[0043] Based on the above solution, it is clear that this embodiment is specifically designed for the common scenario of trunk opening. By receiving a second instruction indicating that the trunk is open, this application can accurately identify the blind spots in the surround view image caused by trunk opening and take corresponding measures to fill these blind spots. This improvement also enhances the accuracy and practicality of the surround view image, providing the driver with more reliable driving assistance information.

[0044] In one alternative implementation, the second camera includes a camera that does not participate in the generation of the panoramic image.

[0045] Based on the above solution, this embodiment utilizes a camera that does not participate in the generation of the surround view image to fill in blind spots, especially when the status information of the first camera includes a malfunction. This improvement expands the sources of image data that can be used to fill in blind spots, increasing the flexibility and accuracy of surround view image generation. By combining data from multiple cameras, this application can more comprehensively cover the environmental information around the vehicle, providing drivers with more comprehensive driving assistance.

[0046] In one alternative implementation, generating a first surround view image of the vehicle at a first moment based on the first image includes:

[0047] Based on the fifth and sixth images, a first surround view image of the vehicle at a first moment is generated. The first image includes the fifth and sixth images. The fifth image is an image captured by at least one second camera before the first moment, and the sixth image is an image captured by at least one second camera at the first moment.

[0048] Based on the above scheme, it can be seen that the first image in this embodiment can be an image captured by at least one second camera. This embodiment realizes the process of generating a panoramic image by combining the first image (including historical images from the second camera and the current image from the second camera).

[0049] By integrating multiple image data, this application can more accurately reflect the actual situation around the vehicle, improving the accuracy and completeness of the surround view image. This improvement provides the driver with a clearer and more intuitive view of the vehicle's surrounding environment, enhancing driving safety and convenience.

[0050] In one optional implementation, the method further includes:

[0051] Output the first panoramic image.

[0052] For example, a first surround view image is displayed on an in-vehicle display.

[0053] Based on the above scheme, this embodiment presents the processed image data intuitively to the driver or other users by outputting a surround view image. This not only ensures the practical application value of the surround view image generation method but also provides the driver with more intuitive and easily understandable driving assistance information.

[0054] Secondly, this application provides a surround view image generation device, which includes: a processing module, used to determine the surround view image blind zone of a vehicle based on the status information of a first camera, wherein the surround view image blind zone is located within the surround view acquisition field of view of the first camera, and the status information of the first camera is used to indicate the pose change of the first camera or the malfunction of the first camera.

[0055] The control module is used to generate a first surround view image of the vehicle at a first moment based on the first image. The first image includes an image captured by a first camera before the first moment and / or an image captured by at least one second camera. The first image is used to fill in the blind spots of the surround view image. The at least one camera also includes at least one second camera.

[0056] In one alternative implementation, where the first image includes an image captured by the first camera before a first moment, and the state information of the first camera is used to indicate changes in the pose of the first camera, and a first surround view image of the vehicle at the first moment is generated based on the first image, the control module is specifically used to generate the first surround view image at the first moment based on the first image and the second image, the second image includes an image captured by the first camera at the first moment, and the field of view corresponding to the second image includes at least a portion of the blind spot of the surround view image.

[0057] In an alternative implementation, in the case where a first surround view image of the vehicle at a first moment is generated based on the first image, the control module is specifically configured to generate the first surround view image at the first moment based on the first image while the vehicle remains in motion.

[0058] In an optional implementation, where the first image includes an image captured by the first camera before a first moment, and a first surround view image of the vehicle at the first moment is generated based on the first image, the control module is specifically used to generate the first surround view image at the first moment based on the third image. The first image includes the third image, which is an image captured by the first camera at a second moment. The second moment is earlier than the first moment and is closest to the first moment. The field of view corresponding to the third image includes the blind spot of the surround view image.

[0059] In an optional implementation, where the first image includes an image captured by the first camera before a first moment, and a first surround view image of the vehicle at the first moment is generated based on the first image, the control module is specifically used to generate the first surround view image at the first moment based on the fourth image. The first image includes the fourth image, which is an image captured by the first camera at a third moment. The third moment is earlier than the moment indicated by the state information of the first camera. The field of view corresponding to the fourth image includes the blind spot of the surround view image.

[0060] In an alternative implementation, in the case where a first surround view image of the vehicle is generated based on the first image at a first moment, the control module is specifically used to determine the position of the first pixel of the first target object in the first image.

[0061] Based on the vehicle's driving status information and the position of the first pixel, the position of the second pixel in the first surround view image of the first target object at the first moment is estimated. The driving status information includes driving distance or driving angle.

[0062] Based on the position of the second pixel, a first surround view image of the vehicle at the first moment is generated.

[0063] In an alternative embodiment, the device further includes a communication module, which is used to acquire the status information of the first camera before determining the blind spot of the vehicle's surround view image based on the status information of the first camera.

[0064] In one optional implementation, the status information of the first camera is used to indicate changes in the pose of the first camera. In the case of obtaining the status information of the first camera, the communication module is specifically used to receive first indication information, which is used to indicate the rearview mirror of the folding vehicle. The first camera includes a side-view camera installed on the rearview mirror of the vehicle.

[0065] In one optional implementation, the status information of the first camera is used to indicate the change in the pose of the first camera and to obtain the status information of the first camera. The communication module is specifically used to receive second instruction information, which is used to instruct the opening of the vehicle's trunk. The first camera includes a rearview camera installed on the trunk cover of the vehicle.

[0066] In one alternative implementation, the second camera includes a camera that does not participate in the generation of the panoramic image.

[0067] In one optional implementation, a first surround view image of the vehicle at a first moment is generated based on the first image. The control module is specifically used to generate the first surround view image of the vehicle at the first moment based on the fifth image and the sixth image. The first image includes the fifth image and the sixth image. The fifth image is an image captured by at least one second camera before the first moment, and the sixth image is an image captured by at least one second camera at the first moment.

[0068] In one alternative implementation, the communication module is also used to output a first surround view image.

[0069] Thirdly, this application provides a vehicle control device, which includes a processor and a memory, wherein the memory is used to store program instructions; the processor calls the program instructions in the memory to cause the vehicle control device to execute the method in the first aspect or any possible implementation of the first aspect.

[0070] Fourthly, this application provides a vehicle including at least one camera, a memory, and a processor. The at least one camera is used to acquire images of the vehicle's surroundings, and the at least one camera includes a first camera. The memory is used to store a computer program, and the processor is used to execute the computer program to implement the method in the first aspect or any possible implementation of the first aspect.

[0071] Fifthly, this application provides a chip including a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface to execute the method in the first aspect or any of the optional embodiments mentioned in the first aspect.

[0072] In a sixth aspect, this application provides a computer-readable storage medium including computer instructions that, when executed by a processor, implement the method in the first aspect or any possible implementation thereof.

[0073] In a seventh aspect, this application provides a computer program product that, when executed by a processor, implements the method described in the first aspect or any possible embodiment of the first aspect.

[0074] For example, the computer program product is a software installation package.

[0075] The technical effects of the second to seventh aspects mentioned above can be referred to the description of the first aspect above, and will not be repeated here. Attached Figure Description

[0076] Figure 1 is a schematic diagram of a vehicle surround view image provided in an embodiment of this application;

[0077] Figure 2 is a schematic diagram of a vehicle surround view blind spot provided in an embodiment of this application;

[0078] Figure 3 is a schematic diagram of a surround view image of a vehicle rearview camera under a change in pose, provided in an embodiment of this application;

[0079] Figure 4 is a schematic diagram of a surround view image of a vehicle in the event of a malfunction of the right-side camera provided in an embodiment of this application;

[0080] Figure 5 is a functional block diagram of a vehicle;

[0081] Figure 6 is a schematic diagram of a vehicle-mounted camera provided in an embodiment of this application;

[0082] Figure 7 is a schematic diagram of another vehicle-mounted camera provided in an embodiment of this application;

[0083] Figure 8 is a schematic diagram of a vehicle interior component provided in an embodiment of this application;

[0084] Figure 9 is a flowchart illustrating a method for generating a panoramic image according to an embodiment of this application;

[0085] Figure 10 is a schematic diagram of an image captured by a first camera according to an embodiment of this application;

[0086] Figure 11 is a schematic diagram of an image captured by a first camera according to another embodiment of this application;

[0087] Figure 12 is a schematic diagram of a first surround view image generation method provided in an embodiment of this application;

[0088] Figure 13 is a schematic diagram of a second moment provided in an embodiment of this application;

[0089] Figure 14 is a schematic diagram of a third moment provided in an embodiment of this application;

[0090] Figure 15 is a schematic diagram of a second camera provided in an embodiment of this application;

[0091] Figure 16 is a schematic diagram of filling the blind area of ​​a ring view image according to an embodiment of this application;

[0092] Figure 17 is a schematic diagram of a surround view image generation device provided in an embodiment of this application;

[0093] Figure 18 is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0094] Currently, vehicles typically have a four-camera surround-view system, meaning that a camera is usually installed at the front, rear, left, and right sides of the vehicle, corresponding to the front-view camera, rear-view camera, left-view camera, and right-view camera. By stitching together the images or videos captured by these four cameras (or the four-camera surround-view system), a bird's-eye view of the vehicle's surroundings can be obtained. Some related technologies also include six-camera surround-view systems. By displaying this bird's-eye view of the vehicle's surround-view video or image to the user, drivers can observe complex road conditions, especially when parking or driving through narrow areas, ensuring safe driving. Furthermore, the vehicle surround-view video or image can also be used to trace the source of accidents in the event of vehicle damage (such as scratches).

[0095] For example, please refer to Figure 1, which is a schematic diagram of a vehicle surround view image provided in an embodiment of this application. As shown in Figure 1, the vehicle surround view image in Figure 1 provides a bird's-eye view of the area around the vehicle. For example, in the side view image, a pillar can be seen on one side of the vehicle, and it is relatively close to the vehicle. In the surround view image, pillars can be seen on both sides of the vehicle, and they are also relatively close to the vehicle. The driver can use the above images to help observe the situation around the vehicle, thereby improving driving safety.

[0096] In practical applications, especially when the camera's status changes, the surround view image generated by the vehicle may be incomplete due to changes in external parameters or poor operating conditions. As shown in Figure 1, the vehicle is currently parking in a very narrow space with pillars on both sides. Therefore, during parking, there is a possibility of the rearview mirrors rubbing or bumping against the pillars. To address this, the driver typically manually adjusts the rearview mirrors to fold them down, preventing rubbing or bumping. Alternatively, in automatic parking scenarios, the automatic parking system will usually proactively fold the rearview mirrors to prevent potential collisions.

[0097] Generally, the left-side and right-side cameras, which capture images of both sides of the vehicle, are mounted on the rearview mirrors. As described above regarding surround-view systems, when the rearview mirrors are folded, the extrinsic parameters of the left-side and right-side cameras change. Furthermore, current algorithms cannot display the correct vehicle surround-view image when the extrinsic parameters change, resulting in blind spots in the vehicle surround-view image. The vehicle surround-view image displayed to the user inside the vehicle might be as shown in Figure 2, a schematic diagram of a vehicle surround-view image blind spot provided in an embodiment of this application. This blind spot is generally represented by a dark color, indicating that the sides of the vehicle body cannot be displayed. The blind spot is specifically caused by changes in the pose (including extrinsic parameters) of the left-side and right-side cameras involved in generating the vehicle surround-view image. This undoubtedly affects the accuracy of the vehicle surround-view image and may potentially pose safety hazards to the vehicle and driver.

[0098] It should be noted that in practical applications, other issues may arise where changes in the state of the cameras involved in generating the vehicle surround view image lead to incomplete display of the surround view image. As shown in Figures 3 and 4, Figure 3 is a schematic diagram of a surround view image under a change in the pose of the vehicle's rear-view camera, provided in an embodiment of this application, and Figure 4 is a schematic diagram of a surround view image under a malfunction of the vehicle's right-view camera, provided in an embodiment of this application.

[0099] In Figure 3, when the rearview camera is mounted on the trunk lid, after the trunk is opened, the rearview camera displays the wall or roof above and behind the vehicle, rather than the ground. In this case, the vehicle surround view image will usually show the rear of the vehicle in a special color or display incorrect splicing information.

[0100] In Figure 4, when the right-view camera malfunctions, the panoramic image is not fully displayed. This is usually indicated by a special color marking the image that the right-view camera should have originally provided.

[0101] It should be noted that the specific representations of the panoramic images in the above-mentioned cases are merely examples. In actual implementations, panoramic images may appear in other image representations, which this application does not limit.

[0102] Based on this, embodiments of this application provide a method, apparatus, and vehicle for generating surround view images, which ensures the integrity and accuracy of surround view images when the state of the vehicle-mounted camera changes, thereby enhancing driving safety and convenience.

[0103] The technical solutions (including methods, devices, and vehicles) in the embodiments of this application will now be described with reference to the accompanying drawings.

[0104] Before describing the technical solution, the possible implementations of the descriptions and terminology that may be involved in this application are clarified as follows:

[0105] In the embodiments of this application, "instruction" can include direct and indirect instructions, as well as explicit and implicit instructions. The information indicated by a certain piece of information is called the information to be instructed. In the specific implementation process, there are many ways to instruct the information to be instructed, such as, but not limited to, directly instructing the information to be instructed, such as the information to be instructed itself or its index. It can also indirectly instruct the information to be instructed by instructing other information, where there is a relationship between the other information and the information to be instructed. It can also instruct only a part of the information to be instructed, while the other parts are known or pre-agreed upon. For example, the instruction of specific information can be achieved by using a pre-agreed (e.g., protocol-defined) arrangement of various pieces of information, thereby reducing instruction overhead to some extent. At the same time, common parts of various pieces of information can be identified and uniformly indicated to reduce the instruction overhead caused by individually indicating the same information.

[0106] Furthermore, the specific indication method can also be any existing indication method, such as, but not limited to, the above-mentioned indication methods and their various combinations. Specific details of various indication methods can be found in existing technologies, and will not be repeated here. As described above, for example, when multiple pieces of information of the same type need to be indicated, the indication methods for different pieces of information may differ. In the specific implementation process, the required indication method can be selected according to specific needs. This application embodiment does not limit the selected indication method; therefore, the indication methods involved in this application embodiment should be understood to cover various methods that enable the party to be indicated to obtain the information to be indicated.

[0107] It should be understood that the information to be indicated can be sent as a whole or divided into multiple sub-information messages sent separately, and the sending period and / or timing of these sub-information messages can be the same or different. The specific sending method is not limited in this application embodiment. The sending period and / or timing of these sub-information messages can be predefined, for example, according to a protocol, or configured by the sending device by sending configuration information to the receiving device.

[0108] "Predefined" or "pre-configured" can be achieved by pre-saving corresponding codes, tables, or other means that can be used to indicate relevant information in the device. This application does not limit the specific implementation method. "Saving" can refer to saving in one or more memories. These memories can be separate installations or integrated into the encoder, decoder, processor, or communication device. Alternatively, some memories can be separately installed, while others are integrated into the decoder, processor, or communication device. The type of memory can be any form of storage medium, and this application does not limit this.

[0109] In the embodiments of this application, descriptions such as "when," "under the circumstances," "if," and "if" all refer to the device making corresponding processing under certain objective circumstances, and are not limited to a specific time. They do not require the device to make a judgment action during implementation, nor do they imply any other limitations.

[0110] In the description of the embodiments of this application, unless otherwise stated, " / " indicates that the objects before and after are in an "or" relationship. For example, A / B can represent A or B. "And / or" in the embodiments of this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. Furthermore, in the description of the embodiments of this application, unless otherwise stated, "multiple" refers to two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple. Additionally, to facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or order of execution, and that "first," "second," etc., are not necessarily different. Furthermore, in the embodiments of this application, words such as "exemplary" or "for example" are used to indicate that something is being used as an example, illustration, or description. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner for ease of understanding.

[0111] The system architecture and business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0112] The specific equipment used in this application will be described below by way of example. For ease of understanding, an architecture of a vehicle to which this application can be applied will be introduced first.

[0113] Please refer to Figure 5, which is a functional block diagram of a vehicle. The vehicle may include various subsystems, such as a motion system 21, a control system 22, a sensing system 23, one or more peripheral devices 24, a power supply 26, and a computing device 101. Each subsystem may include one or more components, and subsystems or components may be interconnected via wired or wireless means. Of course, the functional block diagram shown in Figure 5 is for illustrative purposes only. In actual implementation, the vehicle may include more or fewer subsystems, and the types and numbers of components included in the subsystems may also be designed differently.

[0114] In this application, camera 235 includes at least one camera, which includes a first camera. The first camera is the camera that participates in the generation of the vehicle surround view image. Taking a common 4-camera surround view system as an example, a vehicle typically has one camera installed at the front, rear, left, and right sides, which can be referred to as a front-view camera, a rear-view camera, a left-view camera, and a right-view camera, respectively. By stitching together the images or videos captured by these four cameras (or the 4-camera surround view system), a bird's-eye view video or image of the vehicle's surroundings can be obtained. Here, the front-view camera, rear-view camera, left-view camera, and right-view camera can all be referred to as the first camera. See Figure 6 for details. Figure 6 is a schematic diagram of an in-vehicle camera provided in an embodiment of this application. All cameras shown in Figure 6 can be referred to as first cameras.

[0115] The onboard camera 235, positioned around the vehicle, is a device used to acquire images of the vehicle's surrounding environment. It provides data support for autonomous or assisted driving, or displays images of the surrounding scene to the driver. Currently, to detect distant objects and ensure a wide field of view, multiple focal length cameras are often used. Telephoto cameras have a long detection distance but a narrow field of view, while wide-angle cameras have a short detection distance but a wide field of view, thus complementing each other.

[0116] In one optional embodiment, the first camera is a fisheye camera. Fisheye cameras are generally used to collect real-time image information around a vehicle. Since a fisheye camera is a special ultra-wide-angle lens, its structure mimics the imaging of a fish's eye, allowing for independent wide-angle shooting. The field of view of a fisheye camera can even reach 180°, enabling the capture of images of objects in a wide-area scene. By setting one fisheye camera in each direction around the vehicle, four fisheye cameras can capture a panoramic image of the vehicle's surroundings, thus saving on the number of cameras and reducing costs. Of course, in this embodiment, the camera is not limited to a wide-angle camera; it can also be a regular camera. Increasing the number of regular cameras increases the field of view. For example, setting two or three cameras in each direction around the vehicle and using regular cameras to capture panoramic images of the vehicle's surroundings increases the number of cameras, but the image information captured by regular cameras is undistorted, resulting in better image quality. In this embodiment, a fisheye camera is used as an example, and the number of fisheye cameras is taken as four, i.e., one fisheye camera is set in each direction (front, rear, left, and right) of the vehicle, with the left and right fisheye cameras respectively mounted on the rearview mirrors on both sides of the vehicle.

[0117] In one alternative implementation, the vehicle has other cameras besides the first camera involved in generating the surround view image, such as side-view cameras installed on both sides of the vehicle body, as shown in Figure 7. The side-view cameras in Figure 7 are pinhole cameras used to collect images of the side of the vehicle body at a slightly greater distance from the vehicle body.

[0118] It should be noted that this application does not limit the installation position of the vehicle-mounted camera 235. Considering that the method involved in this application is based on the change of the state information of the first camera, except for the change of the pose of the left and right cameras caused by the folding of the rearview mirror and the change of the pose of the rearview camera caused by the lifting of the trunk lid, when any of the cameras 235 is installed on a movable part such as a door, in the case where the change of the pose of the part causes the change of the camera's pose to affect the generation of the vehicle's surround view image, it can still be determined that the state information of the first camera has changed (including the change of the pose of the first camera or the malfunction of the first camera).

[0119] It should also be noted that the camera 235 can capture video. Correspondingly, when two cameras are capturing video, the images mentioned above refer to image frames in the video.

[0120] The propulsion system 21 includes elements that provide powered motion to the vehicle. Exemplarily, the propulsion system 21 may include an engine 211, an energy source 212, a transmission 213, and wheels 214 (or tires). The engine 211 converts the energy source 212 into mechanical energy and may include an internal combustion engine, an electric motor, an air compressor engine, or other combinations of engines, such as a hybrid engine consisting of a gasoline engine and an electric motor, or a hybrid engine consisting of an internal combustion engine and an air compressor engine. Examples of energy sources 212 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. In some embodiments, the energy source 212 may also provide energy to other systems of the vehicle. The transmission 213 transmits mechanical power from the engine 211 to the wheels 214. The transmission 213 may include a gearbox, a differential, and a drive shaft. Exemplarily, the transmission 213 may also include further elements, such as a clutch.

[0121] Control system 22 is a system for controlling the vehicle and its components. Control system 22 may include various components, including a steering system 221, a throttle 222, or a braking unit 223, etc. Steering system 221 adjusts the vehicle's direction of travel, for example, by including a steering wheel system. Throttle 222 controls the operating speed of engine 211, thereby controlling the vehicle's speed. Braking unit 223 controls vehicle deceleration, and can be used to slow down the rotational speed of wheels 214, thereby controlling vehicle deceleration. In some embodiments, braking unit 223 can convert the kinetic energy of wheels 214 into electrical current. Of course, braking unit 223 may also take other forms to slow down the rotational speed of wheels 214, thereby controlling vehicle speed. Of course, in some embodiments, control system 22 may include more or fewer components, or alternatively, the components shown; for example, control system 22 may also include one or more of a computer vision system, a route control system, an obstacle avoidance system, or an emergency braking system, etc.

[0122] Sensing system 23 may include several sensors that sense information about the environment surrounding or inside the vehicle. For example, sensing system 23 may include a positioning system 231 (which may be a Global Positioning System, BeiDou Navigation Satellite System, or other positioning systems), an inertial measurement unit (IMU) 232, radar 233, lidar 234, a camera 235, a pressure sensor (not shown), a touch sensor (not shown), etc. Sensing system 23 may also include sensors that sense the internal environment of the vehicle (e.g., an in-vehicle air quality monitor, fuel gauge, oil temperature gauge, etc.). Sensor data from one or more of these sensors can be used to detect objects and their corresponding characteristics (position, shape, orientation, speed, etc.).

[0123] The positioning system 231 can be used to obtain the geographical location of the vehicle.

[0124] The IMU232 is used to sense changes in a vehicle's position and orientation based on inertial acceleration. In one embodiment, the IMU232 is a combination of an accelerometer and a gyroscope.

[0125] Radar 233 can use signals such as electromagnetic waves or sound waves to sense objects in the vehicle's surrounding environment and obtain relevant information about the objects. This relevant information may include one or more of the following: distance, angle, speed, direction of travel, reflectivity, color, texture, size, and orientation. LiDAR 234 can use light to sense objects in the vehicle's environment and obtain relevant information about them. Exemplarily, LiDAR 234 may include one or more laser sources, a laser scanner, and one or more detectors, as well as other system components (such as optical elements).

[0126] Camera 235 is used to collect image data, including images and videos. Camera 235 may include a monocular camera, a binocular camera, a time-of-flight (TOF) camera, a camera of a driver monitoring system (DMS), or a camera of a cockpit monitoring system (CMS). The vehicle includes cameras 235 facing the cabin and those facing the surroundings of the vehicle, as shown in Figure 5. Cameras 235 facing the cabin can collect images inside the cabin, while cameras 235 facing the surroundings of the vehicle can collect images around the vehicle. Of course, the number and placement of cameras 235 can be set according to actual needs, such as being placed on the side mirrors, at the front or rear of the vehicle, or near the A-pillar on the driver's side, near the A-pillar on the passenger side, on the headrest (or backrest) of the front seats, at the front of the cabin roof, or integrated into the central control screen.

[0127] Peripheral device 24 provides a means of interaction with the user. Peripheral device 24 may include one or more of the following: communication system 241, screen (or display screen) 242, microphone 243 and / or speaker 244.

[0128] The communication system 241 can communicate with one or more devices directly or via a communication network, including wired and wireless communication.

[0129] Screen 242 can display information to the user. Screen 242 includes one or more of the following: physical screen (such as central control screen 2421), projection system, smart entity, or button panel. Projection system includes, for example, light field screen, head-up display (HUD), or other projection system. As shown in Figure 8, a central control screen 2421 can be installed on the vehicle's center console. The central control screen 2421 is used to display the vehicle's surround view image, driving route, display configuration information of the vehicle's functions, or play audio and video, etc. In addition to the central control screen, other types of screens 242 are also installed in the vehicle, allowing passengers in the front passenger seat, rear seats, etc., to interact with the vehicle through the screen 242. Of course, the screen 242 (including central control screen 2421) here includes one or more of the following: physical screen, projection system, smart entity, or button panel. Projection system includes, for example, light field screen, head-up display (HUD), or other projection system. This application does not strictly limit the number and location of the screens 242 in the applicable scenario. Taking the application of this application in a vehicle as an example, the screens 242 can be located in front of the passenger seat, on the headrest (or backrest) of the front seat, on the armrest of the seat, on the door, or on the top of the cabin.

[0130] Microphone 243 can receive voice commands or other audio input from users inside the vehicle and can be considered a voice acquisition device. As shown in Figure 5, microphone 243 can be installed in the vehicle's steering wheel or integrated into the in-vehicle display device, etc. Microphone 243 can collect sounds inside the cabin, such as user voice commands. The speaker can play audio (including voice prompts, music, sound effects, etc.) to convey information to the user in the form of sound.

[0131] Speaker 244 can output audio to a user inside the vehicle. For example, speaker 244 can perform voice announcements and / or sound effects playback, such as indicating the current status of a controlled object and providing feedback on the execution of an operation. Alternatively, the speaker can play sound effects, such as conveying information by playing a "beep" sound at a specific frequency. In some embodiments, speaker 244 can also be a sound wave emitting device, for example, serving as the transmitter of an ultrasonic detection system (such as ultrasonic radar).

[0132] Motor 245 can generate vibration.

[0133] Power source 26 can provide power to various components of the vehicle. In one embodiment, power source 26 can be a rechargeable lithium-ion or lead-acid battery. One or more such battery packs can be configured to provide power to various components of the vehicle. In some embodiments, power source 26 and energy source 212 can be implemented together, as is the case in some fully electric vehicles.

[0134] Some or all of the vehicle's functions are controlled by computing device 101. Computing device 101 may include at least one processor 1011, which executes instructions 1023 stored in a non-transitory computer-readable medium such as memory 1012. Computing device 101 may also be multiple computing devices controlling individual components or subsystems of the vehicle in a distributed manner. Those skilled in the art will understand that there are many possible designs for the number, location, and integration of the processor, computer, or memory. For example, memory may be a hard disk drive or other storage medium located in a casing different from that of a computer. Therefore, references to processors or computers will be understood to include references to a collection of processors or computers or memories that may or may not operate in parallel. Unlike using a single processor to perform the steps described herein, some components, such as steering components and deceleration components, may each have their own processor, which performs calculations only related to the component's specific function.

[0135] In some solutions, the processor 1011 can be located remotely from the vehicle and be able to transmit information with the vehicle.

[0136] In this application, the processor 1011 can receive images from a first camera and a second camera, execute the image processing method shown in the embodiments of this application to obtain a stitched image, and send the stitched image (i.e., a surround-view stitched image) to the screen 242 so that the screen 242 can display the stitched image. In some embodiments, the processor 1011 can also calculate the distance between the vehicle and each object in the surround-view stitched image.

[0137] In some embodiments, memory 1012 may contain instructions 1023 (e.g., program logic) that can be executed by processor 1011 to perform various functions of the vehicle, including those described above. Memory 1012 may also contain additional instructions, including instructions to send data to, receive data from, interact with, and / or control one or more of the mobility system 21, sensing system 23, control system 22, and peripheral devices 24. In addition to instructions 1023, memory 1012 may also store data such as road maps, route information, vehicle position, direction, speed, and other such vehicle data, as well as other information. This information may be used by the vehicle and computing device 101 during operation in autonomous, semi-autonomous, and / or manual modes.

[0138] It should be understood that the connection relationship between the computing device 101 and the components inside the vehicle is not shown in Figure 5, but in a specific implementation, the computing device 101 may be connected to one or more components inside the vehicle. The following description, in conjunction with Figure 8, uses the vehicle's cockpit as an example to illustrate a system applicable to this application. Referring to Figure 8, the vehicle cockpit includes a computing device 101, a CMS, and a microphone 243. The computing device is a device with computing capabilities; for example, the computing device 101 may include multiple processors, such as an MCU and a SOC. The computing device is connected to the CMS, screen 242, and speaker 244, respectively. The microphone 243 is capable of collecting sound information (considered a voice acquisition device), and can receive user-input voice information and provide it to the computing device 101.

[0139] In some embodiments, the computing device 101 is also connected to one or more controllers, such as an air conditioning controller, a seat controller, an ambient lighting controller, or a vehicle body controller. These controllers can be used to adjust the controlled object; for example, the computing device can output instructions to the air conditioning controller to adjust the air conditioning temperature.

[0140] In some embodiments, the computing device 101 is also connected to one or more interactive devices (or output devices), such as screen 242, speaker 244, ambient lighting controller, etc., which can be used to provide feedback to the user. For example, Figure 8 also illustrates several types of screens, such as the central control screen 2421, the instrument panel screen, and the passenger screen. In specific implementations, the cockpit may include more or fewer types of screens, such as projection screens.

[0141] Alternatively, one or more of the components in the vehicle may be installed or associated separately from the vehicle. For example, the memory 1012 may exist partially or completely separately from the vehicle. The aforementioned components may be communicatively coupled together in a wired and / or wireless manner.

[0142] It should be understood that the functional block diagram shown in Figure 5 is only an example. In actual applications, components in the vehicle can be added or removed according to actual needs. Figure 5 should not be construed as a limitation on the embodiments of this application. The aforementioned vehicle can be a car, truck, motorcycle, bus, ship, airplane, helicopter, lawnmower, recreational vehicle, amusement park vehicle, construction equipment, tram, golf cart, train, or handcart, etc., that has the capability to generate surround view images. The embodiments of this application do not impose any special limitations.

[0143] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0144] Please refer to Figure 9, which is a flowchart illustrating a surround view image generation method provided in an embodiment of this application. This method can be applied to the vehicle shown in Figure 5, or other vehicles including at least one camera. This application will describe the method using the vehicle shown in Figure 5 as an example. The method can also be applied to components within the vehicle. Exemplarily, the method can be executed by the first vehicle shown in Figure 5 or the computing device 101 in the first vehicle. For ease of description, the following description will use a vehicle as the executing entity.

[0145] The method for generating a panoramic image as shown in Figure 9 may include one or more steps from steps S901 to S902. It should be understood that, for ease of description, the description is presented in the order of steps S901 to S902, but this embodiment does not limit the order of execution, the execution time, or the number of executions of the above one or more steps. Steps S901 to S902 are as follows:

[0146] Step S901: Determine the blind spots of the vehicle's surround view image based on the status information of the first camera.

[0147] The first camera is a camera mounted on the vehicle; that is, the vehicle includes the first camera. Optionally, the first camera is a camera involved in generating a surround-view image of the vehicle. For example, multiple cameras are involved in generating the surround-view image of the vehicle, distributed around the vehicle to acquire images of the surrounding area. The first camera can be any one or more of these multiple cameras. For example, as shown in FIG. 7, the first camera can be a camera mounted on a rearview mirror facing one side of the vehicle body, or a camera mounted on the trunk lid facing the rear of the vehicle.

[0148] In one possible implementation, the first camera can be a standalone in-vehicle device, such as adding a first camera to a vehicle without cameras to achieve the solution provided in this application embodiment. Alternatively, existing vehicle cameras, such as front-view cameras (e.g., front-facing dashcams) or rear-view cameras (rear-facing dashcams or rear-facing reversing cameras), can be reused as the first camera, and then used in conjunction with the vehicle infotainment system to achieve the solution provided in this application embodiment. The functions implemented in this application embodiment can also be configured into the vehicle during vehicle production, and this application does not limit this. It should be noted that the method of this application can be applied not only to vehicles with four cameras, but also to vehicles with more cameras.

[0149] Optionally, to adapt to different applications, the lens of the first camera can be of various types, such as zoom lens, fixed-focus lens, fisheye lens, wide-angle lens, telephoto lens, macro lens, pinhole lens, etc., and this application does not limit it. However, for ease of explanation, this application will use a fisheye lens as the lens of the first camera and a fisheye camera as an example for illustration.

[0150] The status information of the first camera is used to indicate changes in the pose of the first camera and / or malfunctions in its operation. Two possible implementations of the first camera's status information are described below.

[0151] Implementation 1: The state information of the first camera is used to indicate the pose changes of the first camera.

[0152] As one possible implementation, the pose change of the first camera includes a change in the extrinsic parameters of the first camera. For example, consider a first camera mounted on the right-side rearview mirror of a vehicle and facing the right side of the vehicle. The extrinsic parameters of the camera include one or more of the following: rotation parameters, which include rotation parameters along three axes, typically denoted as ω, δ, and θ, describing the rotation state of the camera in the world coordinate system; translation parameters, which also include translation parameters along three axes, typically denoted as Tx, Ty, and Tz, describing the position of the camera in the world coordinate system; and a rotation matrix R and a translation vector t, which together describe the pose of the camera in the world coordinate system. When the right-side rearview mirror of the vehicle is folded, one or more of the camera's rotation parameters, translation parameters, rotation matrix R, and translation vector t change, thus changing the extrinsic parameters of the camera, i.e., changing the pose of the camera.

[0153] Optionally, any change in any extrinsic parameter of the first camera can be considered a change in pose.

[0154] In an optional implementation, step S901 may specifically involve determining the blind spot of the vehicle's surround view image when the state information of the vehicle's first camera changes. For example, the method corresponding to FIG9 further includes: determining that the state information of the first camera has changed.

[0155] In one optional implementation, the state information of the first camera can be obtained based on changes in the pose of the corresponding position or component of the first camera. Optionally, the method corresponding to FIG9 further includes: obtaining the state information of the first camera.

[0156] As a possible example, in the case where the state information of the first camera is used to indicate changes in the pose of the first camera, obtaining the state information of the first camera can specifically involve receiving component control information. This component control information is used to instruct adjustments to the vehicle-mounted component where the first camera is located. The vehicle-mounted component could be one or more of the following: a rearview mirror, a trunk, etc. Two possible implementation designs are described below:

[0157] Design 1 involves obtaining the status information of a first camera, including receiving first instruction information (i.e., component control information). The first instruction information is used to instruct the folding of the vehicle's rearview mirror. The first camera includes a side-view camera mounted on the vehicle's rearview mirror. Exemplarily, the first instruction information may be sent by a system related to rearview mirror control after determining that the rearview mirror is to be folded, or it may be sent by a system related to rearview mirror control to the cloud for information related to rearview mirror folding, which is then sent from the cloud to the vehicle. This application does not limit the source of the first instruction information, but it should be clear that the first instruction information is closely related to the action of folding the rearview mirror.

[0158] For example, the time of sending the first indication information may be the time when the rearview mirror is folded. In some possible implementations, after determining that the pose of the first camera has changed, the portion of the surround view image captured before the pose change of the first camera can be determined as a blind spot of the surround view image. For example, the surround view image is a 360° surround view image. Taking a side-view camera mounted on the rearview mirror of a vehicle as an example, the first camera originally provides a 100° portion of the approximately 360° surround view image for generating the surround view image. This 100° portion of the surround view image can be regarded as a blind spot of the surround view image, or it can be the field of view range of the first camera before the change of the state information. See Figure 10 for details. Figure 10 is a schematic diagram of an image captured by a first camera according to an embodiment of this application.

[0159] Optionally, considering that the first camera can still capture images after its pose changes, and that these images can still be used as part of the panoramic image, see Figure 11. Figure 11 is a schematic diagram of an image captured by the first camera according to another embodiment of this application. In Figure 11, after the pose of the first camera changes (the rearview mirror is folded), the image capture range of the first camera is smaller than the image capture range shown in Figure 10 (approximately 100° of the panoramic image). Except for the portion that the front-view camera involved in generating the panoramic image can reach, the actual blind spot caused by the change in the pose of the first camera is shown as the shaded area in Figure 11. In this case, the actual blind spot can be considered as the aforementioned panoramic image blind spot.

[0160] Optionally, by using the external parameters of the first camera before and after the rearview mirror is folded, the blind spot caused by the folding of the rearview mirror can be accurately determined and identified.

[0161] The above design is specifically designed for the common scenario of rearview mirror folding. By receiving the first instruction information indicating that the rearview mirror is folding, it can accurately identify the blind spots in the surround view image caused by the rearview mirror folding and take corresponding measures to fill these blind spots.

[0162] Design 2, obtaining the status information of the first camera includes receiving second instruction information (i.e., component control information), the second instruction information being used to instruct the opening of the vehicle's trunk, and the first camera including a rearview camera mounted on the trunk lid of the vehicle. Exemplarily, the first instruction information may be sent by a system related to rearview camera control or a system related to trunk control after determining that the trunk lid has been lifted, or it may be sent by the aforementioned system to the cloud with information related to the trunk lid being lifted, and then sent from the cloud to the vehicle. This application does not limit the source of the second instruction information, but it should be clear that the second instruction information is closely related to the action of lifting the trunk lid.

[0163] For example, the timing of sending the second instruction information could be the moment when the trunk lid begins to lift.

[0164] For example, in a scenario where the first camera includes a rearview camera mounted on the trunk lid of a vehicle, the blind spot of the surround view image can specifically be the image within the image acquisition range before the trunk lid is lifted, i.e., before the pose of the first camera changes.

[0165] The aforementioned design is specifically tailored to the common scenario of opening the trunk. By receiving a second instruction indicating that the trunk is open, this application can accurately identify blind spots in the surround-view image caused by trunk opening and take corresponding measures to fill these blind spots. This improvement also enhances the accuracy and usability of the surround-view image, providing the driver with more reliable driving assistance information.

[0166] Implementation 2: The status information of the first camera includes the first camera's operational malfunction. This is mainly used to characterize situations where the first camera is malfunctioning, specifically indicating that the first camera cannot capture an image or the image is not captured completely. For details, please refer to Figure 4.

[0167] In summary, the aforementioned blind spots in the surround view are located within the surround view field of view of the first camera. These blind spots may encompass the entire surround view field of view or only a portion thereof. This application does not limit this; solutions for filling the blind spots in different situations can be implemented based on subsequent actions, which will not be elaborated upon here. It should also be noted that, in addition to the blind spots determined based on the state information of the first camera, there are also blind spots inherent to the vehicle itself, such as images of the undercarriage. However, such images are irrelevant to this solution. The above implementation primarily addresses the surround view blind spots caused by the state information of the first camera.

[0168] Step S902: Based on the first image, generate a first surround view image of the vehicle at the first moment.

[0169] First, the first image is explained. The first image is used to fill in the blind spots of the aforementioned panoramic image. For example, the field of view corresponding to the first image includes the aforementioned blind spots of the panoramic image, so that the first panoramic image can be generated at the first moment based on the first image.

[0170] In one alternative implementation, the first image includes an image captured by a first camera before a first moment and / or an image captured by at least one second camera, wherein the at least one camera included in the vehicle also includes the at least one second camera described above.

[0171] It should be noted that "first moment" can refer to the current moment, which can be the moment corresponding to the state information of the first camera. For example, the current moment can be the moment when the pose of the first camera changes or the moment when the first camera malfunctions. For ease of explanation, we will use the current moment as the first moment as an example.

[0172] Therefore, images acquired before the first moment can be used to characterize historical images. Thus, the first image includes historical images from the first camera and / or images acquired by at least one second camera.

[0173] Optionally, the second camera can refer to a camera that does not participate in the generation of the vehicle's surround view image. For example, the second camera can be a camera mounted on the side of the vehicle body facing the front and rear directions of the vehicle body. Alternatively, it can refer to a camera that participates in the generation of the vehicle's surround view image. For example, the second camera can also be a camera facing the front or rear of the vehicle.

[0174] Optionally, the images captured by at least one of the second cameras mentioned above include historical images captured by at least one of the second cameras (i.e., images captured before the first moment), and / or images captured at the first moment.

[0175] Optionally, the first image obtained above can be an image directly captured by the vehicle-mounted camera (first camera and / or second camera), or it can be an image obtained by performing image preprocessing operations after being captured by the vehicle-mounted camera. The image preprocessing includes: image flipping, image rotation, Gaussian filtering, erosion and dilation, etc.

[0176] In summary, the first image can be divided into three cases, specifically including:

[0177] Scenario 1: The first image is the image captured by the first camera before the first moment.

[0178] Scenario 2: The first image is an image captured by at least one second camera.

[0179] Scenario 3: The first image includes an image captured by the first camera before the first moment and an image captured by at least one second camera.

[0180] Understandably, regardless of which of the three scenarios mentioned above, the goal is to obtain the image corresponding to the blind spot in the first moment of the surround view. The specific scenario to be used should also take into account whether the vehicle is in motion.

[0181] Before and after the change in the state information of the first camera, the vehicle is stationary. Therefore, the first image can be the image captured by the first camera before the first moment. The target object in the image can be estimated based on the speed and distance of the target object relative to the vehicle, and then displayed in the first surround view image.

[0182] If the vehicle remains in motion before and after the state information of the first camera changes, then the first image can be any of the three cases mentioned above. For specific implementation details, please refer to the following explanation.

[0183] In addition, the specific implementation of the first image should also refer to the state information of the first camera. The state information of the first camera is used to indicate the change in the pose of the first camera, and the first image may be any of the three situations mentioned above.

[0184] If the status information of the first camera is used in the case of a first camera malfunction, i.e., the first camera is damaged, considering that the vehicle may continue to move, the images captured by the first camera before the first moment can only be used within a short driving distance or a short time before the first camera malfunctions, or they may not be usable. Therefore, in the case of the first camera status information being used in the case of a first camera malfunction, the surround view image generation scheme corresponding to situation two is more suitable.

[0185] Optionally, the first image may include multiple images. For example, in case one, the first camera may capture multiple images at a time before the first moment, and the field of view corresponding to these multiple images includes part or all of the blind spot of the panoramic image.

[0186] Alternatively, the first image may include a single image obtained by fusing multiple frames of images.

[0187] The following section elaborates on the first ring view image generation schemes for the three scenarios described above.

[0188] In scenario one, the reason for using the image captured by the first camera before the first moment as the first image is that the range of images captured by the first camera before the first moment during vehicle movement may include the images that should have been captured when the state of the first camera remained unchanged at the first moment. See Figure 12 for details. Figure 12 is a schematic diagram of a first surround view image generation method provided by an embodiment of this application. In Figure 12, the first cameras are the left-view camera and the right-view camera mounted on the vehicle's rearview mirror. The dashed boxes represent the vehicle's movement trajectory, and the shaded areas represent the images captured by the first cameras before the first moment, i.e., the first image.

[0189] As can be seen from Figures 2, 11, and 12, the image captured by the first camera in Figure 12 before the first moment can cover the blind spot of the surround view image shown in Figure 2. In other words, objects that may affect vehicle driving in the blind spot of the surround view image shown in Figure 2 (such as the pillar in Figure 12) are shown in the image captured by the first camera in Figure 12 before the first moment.

[0190] Furthermore, the field of view corresponding to the image captured by the first camera before the first moment, as shown in Figure 12, includes the actual blind zone shown in Figure 11. Therefore, the first panoramic image at the first moment can be generated based on the image captured by the first camera before the first moment (the first image).

[0191] In summary, Figure 12 illustrates that when the vehicle is in motion, such as in a parking scenario with extremely narrow spaces, the first surround view image can be determined by using the image captured by the first camera during the vehicle's movement before the vehicle's pose information changes.

[0192] To fully utilize resources, in an optional implementation, in scenarios where the pose of the first camera changes, the image captured by the first camera at the moment of pose change can still be used. The image after the pose change is combined with the image before the pose change to fill the blind spot in the surround view image and regenerate the surround view image of the vehicle. This will be explained with reference to Figure 11. Specifically, the first image includes the image captured by the first camera before the first moment. The state information of the first camera is used to indicate the change in the first camera's pose. The generation of the first surround view image of the vehicle at the first moment based on the first image can specifically be achieved by generating the first surround view image at the first moment based on the first image and the second image. The second image includes the image captured by the first camera at the first moment, and the field of view corresponding to the second image includes at least a portion of the surround view image blind spot. If the surround view image blind spot is approximately a 100° panoramic surround view image, then the field of view corresponding to the first image includes the actual blind spot shown in Figure 11, and the field of view corresponding to the second image includes the image outside the actual blind spot in the surround view image blind spot. Therefore, based on the combined action of the first image and the second image, the image corresponding to the surround view image blind spot is generated, thereby generating the first surround view image at the first moment.

[0193] Considering that the first image is a historical image and cannot be directly applied to the generation of the panoramic image, image processing and scene transformation are required before it can be used directly. Specifically, historical image processing methods can include training an artificial neural network for historical image processing, and then using this network to complete the stitching. Other historical image processing methods from related technologies can also be used to stitch the panoramic image; this application does not limit this approach. Based on the above solution, this implementation further refines the process of generating the panoramic image. Especially in the case of camera pose changes, it combines images captured by the first camera before and after the first moment to generate a more accurate panoramic image. This method not only considers the impact of camera pose changes on the field of view but also fully utilizes historical image data, thereby more accurately filling in blind spots and avoiding image stitching errors caused by camera pose changes. This improvement further enhances the accuracy and practicality of the panoramic image.

[0194] Considering the large number of images captured by the first camera before the first moment, to further improve the accuracy of the surround view image generation, the image captured by the first camera at the moment closest to the first moment is used as the basis for generating the first surround view image. In other words, the image captured by the first camera at the moment closest to the first moment can be used as an image to fill the blind spots of the surround view image. In an optional implementation, the first image includes images captured by the first camera before the first moment. Based on the first image, generating the first surround view image of the vehicle at the first moment can specifically be based on a third image, where the first image includes the third image, which is the image captured by the first camera at a second moment. The second moment is earlier than the first moment and closest to the first moment, as shown in Figure 13. Figure 13 is a schematic diagram of a second moment provided by an embodiment of this application. In Figure 13, each dashed box represents a moment before the first moment, with the moment closest to the first moment being the second moment. Referring to Figure 12, it can be seen that the field of view corresponding to the third image captured at the second moment includes the blind spots of the surround view image. In other words, the third image includes the image that the first camera should have captured at the first moment.

[0195] This implementation emphasizes selecting the historical image (third image) closest to the first moment to fill in the blind spot when using historical image data. By reducing the image time span, it not only reduces the amount of computation but also improves the accuracy and timeliness of the surround view image. This feature enables the surround view image to more accurately reflect the actual situation around the vehicle and provides the driver with more reliable driving assistance information.

[0196] In the first optional implementation, the first moment is the current moment, and the second moment can be the moment after the pose of the first camera has changed. Correspondingly, the first image can be the image captured by the first camera after the pose of the first camera has changed, except that compared with the current moment, the first image belongs to the historical image.

[0197] Regarding the selection method for the second moment, a pre-defined correspondence between the vehicle's speed range and the time difference between the second and first moments can be established. For example, the correspondence could include: when the vehicle speed is between 10 km / h and 15 km / h, the time difference between the second and first moments is 0.1 seconds, meaning the selected second moment is 0.1 seconds earlier than the first moment. Thus, at the first moment, based on the vehicle's current speed and the aforementioned correspondence, the time difference between the second and first moments is determined. Then, based on the first moment and the determined time difference, the second moment is determined, thereby obtaining the third image.

[0198] In some implementation scenarios, after the state information of the first camera changes, the image it acquires may not include relevant images of the blind spots in the surround view of the vehicle's current location. For example, in a parking scenario, the first camera is a camera mounted on the trunk lid and facing the rear of the vehicle. Therefore, the first image needs to be refined to ensure that it includes relevant images of the blind spots in the surround view of the vehicle's current location. In an optional implementation, the first image includes images captured by the first camera before a first moment. Specifically, generating the first surround view image of the vehicle at the first moment based on the first image can be based on a fourth image. The first image includes the fourth image, which is an image captured by the first camera at a third moment. The third moment is earlier than the moment indicated by the state information of the first camera, and the field of view corresponding to the fourth image includes the surround view blind spots.

[0199] As shown in Figure 14, the third time can be any time before the first time. Furthermore, the third time can be the time in the image corresponding to the time before the first time that includes the image that should be included in the blind zone of the panoramic image. In other words, as long as the field of view corresponding to the image acquired before the first time includes the blind zone of the panoramic image, then the image is the fourth image.

[0200] This implementation focuses on generating a surround view image from images captured before the camera's state changes (the fourth image). This method is particularly suitable for situations where the field of view is limited due to changes in camera state, such as when the trunk is opened, causing a change in the pose of the first camera. By utilizing image data prior to the state change, this application can more accurately predict and fill in blind spots, thereby improving the completeness and accuracy of the surround view image.

[0201] In scenario two, the first image is an image captured by at least one second camera. This at least one second camera can refer to either a camera that does not participate in generating the vehicle's surround-view image; for example, the second camera can be a camera mounted on the side of the vehicle body facing the front-to-rear direction, such as the side-view pinhole camera shown in Figure 7; or it can refer to a camera that participates in generating the vehicle's surround-view image; for example, the second camera can also be a camera facing the front or rear of the vehicle, such as the front-view camera or rear-view camera shown in Figure 6. In one possible example, see Figure 15, which is a schematic diagram of a second camera provided in an embodiment of this application. The second camera in Figure 15 includes both cameras that do not participate in generating the vehicle's surround-view image and cameras that do participate in generating the vehicle's surround-view image.

[0202] First, we will explain the case where at least one second camera refers to a camera that does not participate in the generation of the vehicle's surround view image. For example, if the field of view of any camera that does not participate in the generation of the vehicle's surround view image includes the surround view image blind spot, the image captured by the corresponding camera can be used to fill the surround view image blind spot and generate a first surround view image.

[0203] This example uses a right-view camera installed on the right side mirror of a vehicle as the first camera, and the status information of the first camera includes any obstruction to its operation. In cases where the right-view camera is damaged and cannot participate in the generation of surround view images, or where the right-view camera is obstructed and the images that can be collected are incomplete, the blind spots of the surround view images are first determined. Then, based on the blind spots of the surround view images, at least one matching second camera is selected. In the above cases, a side-view camera installed on the right side of the vehicle body that can acquire images of the right side of the vehicle body can be used as a second camera to participate in the generation of surround view images.

[0204] Furthermore, considering the situation shown in Figure 7, the side-view camera installed on the right side of the vehicle is a pinhole camera, and pinhole cameras are generally responsible for capturing images of the side of the vehicle that are slightly farther away from the vehicle. Therefore, in an optional embodiment, when there are multiple second cameras, the second cameras include cameras that do not participate in the generation of the vehicle's surround view image and cameras that do participate in the generation of the vehicle's surround view image. For example, referring to Figure 16, Figure 16 is a schematic diagram of filling the blind spot of the surround view image provided by an embodiment of this application. In Figure 16, the first camera is a right-view camera installed on the right rearview mirror of the vehicle, and the status information of the first camera includes the first camera's operation being obstructed, resulting in a right-side blind spot in the surround view image, i.e., the fan-shaped area in Figure 16. Considering that the side-view pinhole camera in the second camera mainly captures images of the side of the vehicle that are slightly farther away from the vehicle, it is also necessary to fill the blind spot with images that are closer to the vehicle.

[0205] For example, while the vehicle is moving forward, the front-view camera (shown in Figure 16 as a front-view fisheye camera) can acquire an image of the closest position on the right side of the vehicle at the first moment. Therefore, by combining the front-view camera (the camera involved in generating the surround view image) and the side-view cameras (the cameras not involved in generating the surround view image), the blind spot caused by the malfunction of the right-view camera can be filled, thereby realizing the generation of the first surround view image at the first moment. It should be noted that if the vehicle is moving backward, i.e., during reversing, the front-view camera cannot capture the image of the closest position on the right side of the vehicle at the first moment. Accordingly, other cameras, such as the rear-view camera, are required to capture this image.

[0206] This application does not limit the number of second cameras or their actual installation location; the specific requirements can be determined based on actual needs.

[0207] Furthermore, considering that the images captured by the forward-looking camera, which are closer to the vehicle body, are historical images, and the side-view pinhole cameras mainly capture images of the sides of the vehicle body at a slightly greater distance, the aforementioned generation of the first surround-view image of the vehicle at the first moment based on the first image can specifically be based on the fifth and sixth images. The first image includes the fifth and sixth images, where the fifth image is an image captured by at least one second camera before the first moment, and the sixth image is an image captured by at least one second camera at the first moment. By integrating multiple image data, this application can more accurately reflect the actual situation around the vehicle, improving the accuracy and completeness of the surround-view image. This improvement provides the driver with a clearer and more intuitive view of the vehicle's surrounding environment, enhancing driving safety and convenience.

[0208] In scenario three, based on the aforementioned implementation methods for scenarios one and two, it can be understood that the key to generating the first surround view image in this application embodiment lies in generating the current surround view image using a first image whose field of view includes the blind spot of the surround view image. In some implementation scenarios, there are implementation methods that determine the historical image of the first camera and the current image of at least one second camera as the first image. For example, in an automatic parking scenario, such as a parallel parking scenario, the vehicle at the second moment is not at the same horizontal line as the first moment, but may be in a tilted state. Here, the tilted state is used to characterize that the vehicle itself is tilted relative to the parking space.

[0209] In this implementation scenario, the historical images of the first camera include part of the 360-degree view blind spot, and the current images of at least one second camera include the remaining part of the 360-degree view blind spot. When the two are combined, the corresponding field of view includes the complete 360-degree view blind spot, thus providing the basis for realizing the first 360-degree view.

[0210] In summary, it is crucial to determine whether the field of view of a particular image includes the blind spot of the panoramic image. The following is an example of how to select the first image.

[0211] The selection of the first image can be determined by analyzing images captured by a first camera and / or at least one second camera. For example, after the first camera captures an image at each moment, the vehicle extracts the features of the target object from the image captured by the first camera. When the state information of the first camera changes, i.e., when the pose of the first camera changes or when the first camera encounters an obstacle, the blind spot of the surround view image is determined, and then the target object in the surround view image blind spot is determined. The features of the target object in the image captured by the first camera are compared with the features of the target object in the surround view image blind spot to determine whether the image captured by the first camera includes the target object in the surround view image blind spot.

[0212] The image captured by the first camera includes the target object in the blind spot of the panoramic image, which is considered as case one, that is, the first image is the image captured by the first camera before the first moment.

[0213] If the image captured by the first camera does not include the target object in the blind zone of the panoramic image, the feature of the target object in the image captured by the second camera is extracted and then compared. If the image captured by at least one second camera includes the target object, it is considered as case two, that is, the first image is an image captured by at least one second camera.

[0214] If the image captured by the first camera includes some target objects in the blind zone of the panoramic image, feature extraction is performed on the remaining target objects (the remaining part of the target objects other than the aforementioned target objects) in the image captured by the second camera, and then comparison is performed. If the images captured by one or more second cameras include the aforementioned remaining target objects, it is considered as case three, that is, the first image includes the image captured by the first camera before the first moment and the image captured by at least one second camera.

[0215] Optionally, the target object can refer to pedestrians, buildings, or other objects that may collide with the vehicle.

[0216] The following section explains how to transform historical images.

[0217] In one optional implementation, the position of a first pixel of a first target object in a first image is determined; based on the vehicle's driving status information and the position of the first pixel, the position of a second pixel of the first target object in the first surround view image at a first moment is estimated, the driving status information including driving distance or driving angle; combining the vehicle's driving distance, driving angle and other driving status information, as well as the position information of the target object in the image, the position of the object in the historical image to be changed compared to the current moment is estimated, thereby obtaining the surround view image.

[0218] Optionally, the above embodiments further include determining a first target object in the first image. For example, the first target object may be an object with obvious features other than the ground, such as a parking line, a support post, a bollard, a parked vehicle, etc.

[0219] Furthermore, based on the position of the second pixel, a first surround view image of the vehicle at the first moment is generated.

[0220] The following section explains how to generate the panoramic image.

[0221] In one alternative implementation, the portion of the first image corresponding to the blind spot of the surround view image is stitched together with images captured by other cameras involved in generating the surround view image.

[0222] Specifically, the stitching method can involve training an artificial neural network for surround view stitching, and then using this network to complete the stitching. Alternatively, other surround view stitching methods from related technologies can be used; this application does not limit the specific surround view stitching method.

[0223] Furthermore, after performing the above steps, the first surround-view image at the first moment can also be displayed / output. For example, during vehicle operation, the stitched image at the first moment can be displayed on the vehicle's display screen. If the method shown in this application is executed cyclically, that is, steps S901 and S902 are executed at each moment, a video composed of the vehicle's first surround-view image can also be displayed on the vehicle's display screen, so that the driver can observe the road conditions around the vehicle and ensure driving safety.

[0224] In summary, the method provided in this application improves the completeness and accuracy of the surround view image by comprehensively considering the state information of at least one camera on the vehicle (such as pose changes, operational malfunctions, etc.) and combining historical images with image data from other cameras. Furthermore, this method can not only provide accurate surround view images when the vehicle is stationary, but also generate clear, blind-spot-free surround view images in real time when the vehicle is in motion. Even further, this method creatively introduces camera data that is not involved in the surround view image generation to fill in blind spots. This improvement expands the sources of image data that can be used to fill in blind spots, increasing the flexibility and adaptability of surround view image generation. By combining data from multiple cameras, it is possible to more comprehensively cover the environmental information around the vehicle, providing the driver with more comprehensive driving assistance.

[0225] The foregoing has described the application scenarios and methods provided by the embodiments of this application. The apparatus of the embodiments of this application is provided below. It is understood that the various apparatuses provided in the embodiments of this application, such as surround view image generation apparatuses, computing devices, chips, etc., include hardware structures, software units, or combinations of hardware and software structures to perform the functions described in the above method embodiments. Those skilled in the art should readily recognize that the apparatus and modules within it can be implemented in hardware or a combination of hardware and computer software, in conjunction with the various functions described in the embodiments disclosed herein. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different apparatus implementation methods to implement the aforementioned method embodiments in different application scenarios, and different implementation methods of the apparatus should not be considered beyond the scope of the embodiments of this application.

[0226] Several possible devices are listed below.

[0227] Please refer to Figure 17, which is a schematic diagram of a surround view image generation device provided in an embodiment of this application. The surround view image generation device 170 may include a processing module 1701, a control module 1702, and a communication module 1703. The surround view image generation device 170 can be a standalone device, such as the computing device 101 shown in Figure 5 or the first vehicle shown in Figure 5. Alternatively, the surround view image generation device 170 can also be a software module and / or hardware module in a standalone device, such as a chip in the computing device 101.

[0228] The surround view image generation device 170 is used to implement the aforementioned surround view image generation method, such as the surround view image generation method in the embodiment shown in FIG9. The processing module is used to implement one or more of the aforementioned data processing, instruction execution, and other related operations such as determining, adjusting, generating, deciding, and judging; the control module is used to implement the aforementioned operations such as generating surround view images; and the communication module is used to implement one or more of the aforementioned operations such as acquiring and receiving. Optionally, the surround view image generation device further includes a feedback module 1704, which is used to provide feedback information to the user, such as outputting prompts or outputting the first surround view image at the first moment.

[0229] The panoramic image generation device 170 can realize corresponding functions based on the methods and system-related content described above. For specific operation, please refer to the description in the embodiment shown in Figure 9. This application will not elaborate further on this.

[0230] Referring to Figure 18, which is a schematic diagram of the structure of a computing device provided in an embodiment of this application, the computing device 101 is a device with computing capabilities. This device can be a physical device, such as a controller, processor, server (e.g., rack server), host, etc., or it can be a virtual device, such as a virtual machine, container, etc. Optionally, the computing device 101 can be included in a vehicle, as shown in Figure 5, where the computing device 101 is included within a first vehicle. Optionally, the computing device 101 is the aforementioned vehicle control device.

[0231] As shown in Figure 18, the computing device 101 includes a processor 1011 and a memory 1012. Optionally, the computing device 101 may also include one or more of a bus 1015, a communication interface 1014, etc. For example, the processor 1011 and the memory 1012 communicate via the bus 1015. It should be understood that this application does not limit the number of processors and memories in the computing device 101.

[0232] The memory 1012 provides storage space, which may optionally store application data, user data, operating system, and computer programs (including the aforementioned instructions 1023). The memory 1012 may include volatile memory, such as random access memory (RAM). The memory 1012 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD).

[0233] Processor 1011 is a module that performs computations and may include any one or more of the following: controller (e.g., memory controller), central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), digital signal processor (DSP), coprocessor (to assist the central processing unit in performing corresponding processing and applications), application-specific integrated circuit (ASIC), microcontroller unit (MCU), virtual machine, container, etc.

[0234] The communication interface 1014 is used to provide information input or output to at least one processor, such as an in-line interface, an out-line interface, etc.

[0235] And / or, the communication interface 1014 can be used to receive data transmitted externally and / or transmit data externally. The communication interface 1014 can be a wired link interface, such as an Ethernet cable, or a wireless link interface (Wi-Fi, Bluetooth, general wireless transmission, and other wireless communication technologies, etc.). Optionally, the communication interface 1014 may also include a transmitter (such as a radio frequency transmitter, antenna, etc.) or a receiver coupled to the interface.

[0236] Bus 1015 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 18, but this does not imply that there is only one bus or one type of bus. Bus 1015 can include pathways for transmitting information between various components of computing device 101 (e.g., memory 1012, processor 1011, communication interface 1014).

[0237] In one possible implementation, memory 1012 stores executable instructions, and processor 1011 executes these executable instructions to implement the aforementioned interactive method, such as the surround view image generation method in the embodiment shown in FIG7.

[0238] This application also provides a chip, including a processor and a communication interface. The communication interface is used for outputting and / or outputting data (including instructions), and / or for receiving and / or sending data. When the processor executes program instructions in memory, the aforementioned interaction method, such as the surround view image generation method in the embodiment shown in FIG9, is used.

[0239] As one possible example, the communication interface is used to input voice and gesture information, and the processor is used to determine the controlled object based on the voice information and adjust the state of the controlled object based on the gesture information. Optionally, the communication interface is also used to output at least some information related to various prompts.

[0240] This application provides a computer-readable storage medium storing instructions that, when executed by at least one processor, implement the aforementioned interactive method, such as the surround view image generation method in the embodiment shown in FIG9.

[0241] The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center that contains one or more available media. Computer-readable storage media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives).

[0242] This application provides a computer program product including computer instructions that, when executed on at least one processor, implement the aforementioned interactive method, such as the surround view image generation method in the embodiment shown in FIG9.

[0243] Optionally, the computer program product can be a software installation package or an image package. If the aforementioned method is required, the computer program product can be downloaded and executed on a computing device.

[0244] This application provides a vehicle that includes the aforementioned surround view image generation device 170, or includes the aforementioned computing device 101, or includes the aforementioned chip, or includes the aforementioned computer storage medium, or deploys the aforementioned computer program product.

[0245] For example, the architecture of the vehicle can be shown in Figure 5.

[0246] In this application, the terms "exemplarily" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplarily" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the terms "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0247] In this application, "at least one" in the embodiments refers to one or more items, and "more than one" refers to two or more items. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent: a, b, c, (a and b), (a and c), (b and c), or (a and b and c), where a, b, and c can be single or multiple. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0248] Furthermore, unless otherwise stated, the use of ordinal numbers such as "first" and "second" in the embodiments of this application is for distinguishing multiple objects and is not for limiting the order, sequence, priority or importance of multiple objects.

Claims

1. A surround view image generation method, characterized by, The method is applied to a vehicle, the vehicle including at least one camera, the at least one camera being used to capture images of the surroundings of the vehicle, the at least one camera including a first camera; The method includes: Based on the status information of the first camera, the surround view blind spot of the vehicle is determined. The surround view blind spot is located within the surround view acquisition field of the first camera. The status information of the first camera is used to indicate the pose change of the first camera or the malfunction of the first camera. Based on the first image, a first surround view image of the vehicle at a first moment is generated. The first image includes an image captured by the first camera before the first moment and / or an image captured by at least one second camera. The first image is used to fill in the blind spots of the surround view image. The at least one camera also includes the at least one second camera.

2. The method of claim 1, wherein, The first image includes an image captured by the first camera before a first moment, and the state information of the first camera is used to indicate changes in the pose of the first camera. The step of generating a first surround-view image of the vehicle at the first moment based on the first image includes: Based on the first image and the second image, a first panoramic image at the first moment is generated. The second image includes an image captured by the first camera at the first moment, and the field of view corresponding to the second image includes at least a portion of the blind zone of the panoramic image.

3. The method according to claim 1 or 2, characterized in that, The step of generating a first surround view image of the vehicle at a first moment based on the first image includes: While the vehicle remains in motion, a first surround view image at the first moment is generated based on the first image.

4. The method according to any one of claims 1 to 3, characterized in that, The first image includes an image captured by the first camera before a first moment. Generating a first surround-view image of the vehicle at the first moment based on the first image includes: Based on the third image, a first panoramic image at the first moment is generated. The first image includes the third image, which is an image captured by the first camera at a second moment. The second moment is earlier than the first moment and is closest to the first moment. The field of view corresponding to the third image includes the blind zone of the panoramic image.

5. The method according to any one of claims 1 to 3, characterized in that, The first image includes an image captured by the first camera before a first moment. Generating a first surround-view image of the vehicle at the first moment based on the first image includes: Based on the fourth image, a first panoramic image at the first moment is generated. The first image includes the fourth image, which is an image captured by the first camera at a third moment. The third moment is earlier than the moment indicated by the state information of the first camera. The field of view corresponding to the fourth image includes the blind zone of the panoramic image.

6. The method according to any one of claims 1 to 5, characterized in that, The step of generating a first surround view image of the vehicle at a first moment based on the first image includes: Determine the position of the first pixel of the first target object in the first image; Based on the vehicle's driving status information and the position of the first pixel, the position of the second pixel in the first surround view image of the first target object at the first moment is estimated. The driving status information includes driving distance or driving angle. Based on the position of the second pixel, a first surround view image of the vehicle at the first moment is generated.

7. The method according to any one of claims 1 to 6, characterized in that, Before determining the blind spot of the vehicle's surround view image based on the state information of the first camera, the method further includes: Obtain the status information of the first camera.

8. The method of claim 7, wherein, The state information of the first camera is used to indicate changes in the pose of the first camera. Obtaining the state information of the first camera includes: The system receives a first instruction message, which instructs the folding of the vehicle's rearview mirror. The first camera includes a side-view camera mounted on the vehicle's rearview mirror.

9. The method of claim 7, wherein, The state information of the first camera is used to indicate changes in the pose of the first camera. Obtaining the state information of the first camera includes: The system receives a second instruction, which instructs the opening of the vehicle's trunk. The first camera includes a rearview camera mounted on the trunk lid of the vehicle.

10. The method according to any one of claims 1 to 9, characterized in that, The second camera includes cameras that do not participate in the generation of the surround view image.

11. The method of claim 10, wherein, The step of generating a first surround view image of the vehicle at a first moment based on the first image includes: Based on the fifth and sixth images, a first surround view image of the vehicle at a first moment is generated. The first image includes the fifth and sixth images. The fifth image is an image captured by at least one second camera before the first moment, and the sixth image is an image captured by at least one second camera at the first moment.

12. The method according to any one of claims 1 to 11, characterized in that, The method further includes: Output the first circular view image.

13. An around view image generation apparatus, characterized by comprising: The device includes: The processing module is used to determine the vehicle's surround view blind spot based on the status information of the first camera. The surround view blind spot is located within the surround view acquisition field of view of the first camera. The status information of the first camera is used to indicate the pose change of the first camera or the malfunction of the first camera. The control module is used to generate a first surround view image of the vehicle at a first moment based on a first image. The first image includes an image captured by the first camera before the first moment and / or an image captured by at least one second camera. The first image is used to fill in the blind spots of the surround view image. The at least one camera also includes the at least one second camera.

14. An around view image generation device, characterized by, The aforementioned device includes a processor and a memory, wherein the memory is used to store program instructions; the processor invokes the program instructions in the memory to cause the surround view image generation device to implement the surround view image generation method according to any one of claims 1-12.

15. A vehicle characterized by comprising: The vehicle includes at least one camera, a memory, and a processor. The at least one camera is used to acquire images of the surroundings of the vehicle. The at least one camera includes a first camera. The memory is used to store a computer program, and the processor is used to execute the computer program to implement the surround view image generation method according to any one of claims 1-12.

16. The vehicle of claim 15, wherein, The at least one camera also includes at least one second camera.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program for implementing the panoramic image generation method according to any one of claims 1-12.

18. A computer program product, characterised in that, The computer program product includes a computer program / instruction that, when executed by a processor, implements the panoramic image generation method according to any one of claims 1-12.

Images